[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:12 dist-test-slave-pp80 krb5kdc[1244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:12 dist-test-slave-pp80 krb5kdc[1244](info): set up 2 sockets
Sep 27 22:40:12 dist-test-slave-pp80 krb5kdc[1244](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:40:13.893927 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:14 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:14.078128 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:43947
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:14.421013 1257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:14.421309 1257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:14.421397 1257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:14.428776 1257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:14.428932 1257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:14.429054 1257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:14.429152 1257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:14.429253 1257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:14.439875 1257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:43947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:14.441303 1257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:14.442688 1257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:40:14.456143 1257 server_base.cc:1034] running on GCE node
W20260927 22:40:14.456195 1262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:14.456101 1265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:14.456144 1263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:14.457315 1257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:14.458776 1257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:14.460060 1257 hybrid_clock.cc:648] HybridClock initialized: now 1790548814459958 us; error 86 us; skew 500 ppm
I20260927 22:40:14.460505 1257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:14 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548814, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:14.463601 1257 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:14.465224 1257 webserver.cc:460] Webserver started at http://127.1.53.126:36457/ using document root <none> and password file <none>
I20260927 22:40:14.465924 1257 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:14.466084 1257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:14.466403 1257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:14.469403 1257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "64d42f24df734b6db9920d573aaec15c"
format_stamp: "Formatted at 2026-09-27 22:40:14 on dist-test-slave-pp80"
server_key: "8031fca06f32dbd2e87090f2f94d4422"
server_key_iv: "e12ccd88629e449a290b5508c74fcf33"
server_key_version: "encryptionkey@0"
I20260927 22:40:14.470134 1257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "64d42f24df734b6db9920d573aaec15c"
format_stamp: "Formatted at 2026-09-27 22:40:14 on dist-test-slave-pp80"
server_key: "8031fca06f32dbd2e87090f2f94d4422"
server_key_iv: "e12ccd88629e449a290b5508c74fcf33"
server_key_version: "encryptionkey@0"
I20260927 22:40:14.475193 1257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:40:14.478854 1272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:14.480159 1257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:40:14.480382 1257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "64d42f24df734b6db9920d573aaec15c"
format_stamp: "Formatted at 2026-09-27 22:40:14 on dist-test-slave-pp80"
server_key: "8031fca06f32dbd2e87090f2f94d4422"
server_key_iv: "e12ccd88629e449a290b5508c74fcf33"
server_key_version: "encryptionkey@0"
I20260927 22:40:14.480645 1257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:14.493356 1257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:14.497308 1257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:14.497608 1257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:14.517426 1257 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:43947
I20260927 22:40:14.517460 1323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:43947 every 8 connection(s)
I20260927 22:40:14.518806 1257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:14.522495 1324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:14.526635 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 1257
I20260927 22:40:14.526921 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:14.528853 1237 external_mini_cluster.cc:1428] Setting key aa1bd68a4518f1f8c25abad8d3676e08
I20260927 22:40:14.533679 1324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c: Bootstrap starting.
I20260927 22:40:14.536767 1324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:14.538005 1324 log.cc:824] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:14.540715 1324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c: No bootstrap required, opened a new log
I20260927 22:40:14.549296 1324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d42f24df734b6db9920d573aaec15c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43947 } }
I20260927 22:40:14.549691 1324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:14.549824 1324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d42f24df734b6db9920d573aaec15c, State: Initialized, Role: FOLLOWER
I20260927 22:40:14.550422 1324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d42f24df734b6db9920d573aaec15c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43947 } }
I20260927 22:40:14.550647 1324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:14.550781 1324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:14.550961 1324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:14.553160 1324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d42f24df734b6db9920d573aaec15c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43947 } }
I20260927 22:40:14.553673 1324 leader_election.cc:302] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d42f24df734b6db9920d573aaec15c; no voters:
I20260927 22:40:14.554383 1324 leader_election.cc:288] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:14.554476 1329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:14.555737 1329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [term 1 LEADER]: Becoming Leader. State: Replica: 64d42f24df734b6db9920d573aaec15c, State: Running, Role: LEADER
I20260927 22:40:14.556326 1329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d42f24df734b6db9920d573aaec15c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43947 } }
I20260927 22:40:14.556751 1324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 22:40:14 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:14.559834 1330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d42f24df734b6db9920d573aaec15c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d42f24df734b6db9920d573aaec15c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43947 } } }
I20260927 22:40:14.560319 1330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:14.561257 1331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d42f24df734b6db9920d573aaec15c. Latest consensus state: current_term: 1 leader_uuid: "64d42f24df734b6db9920d573aaec15c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d42f24df734b6db9920d573aaec15c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43947 } } }
I20260927 22:40:14.561561 1331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:14.563351 1334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:14.573441 1334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:14.577828 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:14.532629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.1.53.126:43947)
0927 22:40:14.533376 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:14.533387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:14.534652 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 22:40:14.545269 (+ 10617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:14.545290 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:14.545678 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:14.546159 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:14.549518 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:14.551359 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:14.553082 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:14.553966 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:14.554179 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:14.562192 (+ 8013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:14.562211 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:14.562240 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:14.562309 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:14.570095 (+ 7786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:14.571355 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:14.571367 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:14.571378 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:14.571651 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:14.572278 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:14.572283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:14.572287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:14.572983 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:14.575647 (+ 2664us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:14.575911 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:40:14.576402 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":344,"threads_started":1}
I20260927 22:40:14.585074 1334 catalog_manager.cc:1347] Generated new cluster ID: 3c3ad16feeb64647aa64332f5b055303
I20260927 22:40:14.585289 1334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:14.602773 1334 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:14.604265 1334 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:14.616168 1334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d42f24df734b6db9920d573aaec15c: Generated new TSK 0
I20260927 22:40:14.617105 1334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:14.665326 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:43947
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:15.007694 1350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:15.008100 1350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:15.008293 1350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:15.015743 1350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:15.015921 1350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:15.016110 1350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:15.026108 1350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:43947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:15.027468 1350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:15.028898 1350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:15.043531 1358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:15.043766 1355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:15.043905 1356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:15.044943 1350 server_base.cc:1034] running on GCE node
I20260927 22:40:15.045578 1350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:15.046598 1350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:15.047843 1350 hybrid_clock.cc:648] HybridClock initialized: now 1790548815047771 us; error 79 us; skew 500 ppm
I20260927 22:40:15.048202 1350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:15 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:15.060022 1350 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:15.061758 1350 webserver.cc:460] Webserver started at http://127.1.53.65:37653/ using document root <none> and password file <none>
I20260927 22:40:15.062613 1350 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:15.062723 1350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:15.063110 1350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:15.066212 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "185795fb860e4bccb26dc29d7aadaa8e"
format_stamp: "Formatted at 2026-09-27 22:40:15 on dist-test-slave-pp80"
server_key: "821625ed65349f5548f7c7516cc56955"
server_key_iv: "601833d07f9b837c8f35f8ca136db668"
server_key_version: "encryptionkey@0"
I20260927 22:40:15.067003 1350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "185795fb860e4bccb26dc29d7aadaa8e"
format_stamp: "Formatted at 2026-09-27 22:40:15 on dist-test-slave-pp80"
server_key: "821625ed65349f5548f7c7516cc56955"
server_key_iv: "601833d07f9b837c8f35f8ca136db668"
server_key_version: "encryptionkey@0"
I20260927 22:40:15.072613 1350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:40:15.076524 1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:15.078106 1350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:15.078284 1350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "185795fb860e4bccb26dc29d7aadaa8e"
format_stamp: "Formatted at 2026-09-27 22:40:15 on dist-test-slave-pp80"
server_key: "821625ed65349f5548f7c7516cc56955"
server_key_iv: "601833d07f9b837c8f35f8ca136db668"
server_key_version: "encryptionkey@0"
I20260927 22:40:15.078504 1350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:15.110214 1350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:15.114743 1350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:15.115090 1350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:15.116186 1350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:15.117808 1350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:15.117897 1350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:15.117982 1350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:15.118069 1350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:15.155015 1350 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:35903
I20260927 22:40:15.155098 1477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:35903 every 8 connection(s)
I20260927 22:40:15.156397 1350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:15.164052 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 1350
I20260927 22:40:15.164340 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:15.164927 1237 external_mini_cluster.cc:1428] Setting key a83c0fc74f1eb57f62dded7b46ef437f
Sep 27 22:40:15 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:15.178367 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:15.159201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:47521 (local address 127.1.53.126:43947)
0927 22:40:15.159666 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:15.159672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:15.161179 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 22:40:15.163135 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:15.163139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:15.163204 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:15.163307 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:15.165195 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.166126 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.167859 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.168357 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.168503 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:15.172540 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:15.172545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:15.172549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:15.172582 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:15.175180 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:15.175951 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:15.175962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:15.175965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:15.176078 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:15.176560 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:15.176565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:15.176568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:15.176837 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:15.176994 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:15.177764 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 22:40:15.177977 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260927 22:40:15.179375 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:15.159769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:43947 (local address 127.1.53.65:47521)
0927 22:40:15.161010 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:15.161057 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:15.162856 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:15.163556 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:15.163565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:15.164153 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:15.164981 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:15.165000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.166290 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.166296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:15.167691 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:15.167703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.168619 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.168627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:15.168886 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:15.169643 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:15.169667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:15.172220 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:15.175345 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:15.175355 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:15.175374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:15.175803 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:15.176193 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:15.176198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:15.176201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:15.176434 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:15.176970 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:15.176979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:15.177383 (+ 404us) client_negotiation.cc:769] Sending connection context
0927 22:40:15.177664 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 22:40:15.178212 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":790,"threads_started":1}
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:40:15.181048 1478 heartbeater.cc:347] Connected to a master server at 127.1.53.126:43947
I20260927 22:40:15.181372 1478 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:15.182111 1478 heartbeater.cc:511] Master 127.1.53.126:43947 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:40:15.184674 1289 ts_manager.cc:194] Registered new tserver with Master: 185795fb860e4bccb26dc29d7aadaa8e (127.1.53.65:35903)
I20260927 22:40:15.186306 1289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:47521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:15.209084 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:43947
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:15.550503 1484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:15.550776 1484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:15.550869 1484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:15.558763 1484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:15.558881 1484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:15.559011 1484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:15.568934 1484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:43947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:15.570179 1484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:15.571457 1484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:15.585624 1489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:15.585767 1492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:15.585788 1490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:15.586665 1484 server_base.cc:1034] running on GCE node
I20260927 22:40:15.587373 1484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:15.588465 1484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:15.589684 1484 hybrid_clock.cc:648] HybridClock initialized: now 1790548815589615 us; error 59 us; skew 500 ppm
I20260927 22:40:15.590037 1484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:15 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:15.593284 1484 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:15.594753 1484 webserver.cc:460] Webserver started at http://127.1.53.66:35913/ using document root <none> and password file <none>
I20260927 22:40:15.595391 1484 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:15.595489 1484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:15.595817 1484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:15.598649 1484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "95a4a4577978412987bc76f30e19c63f"
format_stamp: "Formatted at 2026-09-27 22:40:15 on dist-test-slave-pp80"
server_key: "1a5875c1273fd7a66d5c0867ef9ee2ee"
server_key_iv: "82b4f3247314d0fcf239bde4dd6beb1c"
server_key_version: "encryptionkey@0"
I20260927 22:40:15.599339 1484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "95a4a4577978412987bc76f30e19c63f"
format_stamp: "Formatted at 2026-09-27 22:40:15 on dist-test-slave-pp80"
server_key: "1a5875c1273fd7a66d5c0867ef9ee2ee"
server_key_iv: "82b4f3247314d0fcf239bde4dd6beb1c"
server_key_version: "encryptionkey@0"
I20260927 22:40:15.604417 1484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:40:15.608376 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:15.609874 1484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:15.610059 1484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "95a4a4577978412987bc76f30e19c63f"
format_stamp: "Formatted at 2026-09-27 22:40:15 on dist-test-slave-pp80"
server_key: "1a5875c1273fd7a66d5c0867ef9ee2ee"
server_key_iv: "82b4f3247314d0fcf239bde4dd6beb1c"
server_key_version: "encryptionkey@0"
I20260927 22:40:15.610262 1484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:15.625463 1484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:15.630188 1484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:15.630530 1484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:15.631740 1484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:15.633315 1484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:15.633436 1484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:15.633611 1484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:15.633697 1484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:15.673231 1484 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:40593
I20260927 22:40:15.673324 1611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:40593 every 8 connection(s)
I20260927 22:40:15.674597 1484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:15.677476 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 1484
I20260927 22:40:15.677659 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:15.678149 1237 external_mini_cluster.cc:1428] Setting key 30725feb0d15fd8c4776224dc5b4c8c4
Sep 27 22:40:15 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
I20260927 22:40:15.697438 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:15.677378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:55739 (local address 127.1.53.126:43947)
0927 22:40:15.677592 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:15.677597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:15.679190 (+ 1593us) server_negotiation.cc:407] Connection header received
0927 22:40:15.681344 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:15.681349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:15.681439 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:15.681550 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:15.682999 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.683802 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.686381 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.686817 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.686922 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:15.691837 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:15.691842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:15.691845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:15.691877 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:15.694477 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:15.695224 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:15.695235 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:15.695237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:15.695351 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:15.695796 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:15.695800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:15.695804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:15.696138 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:15.696346 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:15.697012 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 22:40:15.697193 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
I20260927 22:40:15.698602 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:15.678039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:43947 (local address 127.1.53.66:55739)
0927 22:40:15.679010 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:15.679067 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:15.681080 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:15.681792 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:15.681801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:15.682293 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:15.682833 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:15.682848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.683960 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.683965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:15.686230 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:15.686247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:15.687053 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:15.687062 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:15.687387 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:15.688373 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:15.688406 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:15.691585 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:15.694640 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:15.694650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:15.694669 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:15.695088 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:15.695464 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:15.695470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:15.695473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:15.695673 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:15.696324 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:15.696330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:15.696660 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 22:40:15.696888 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 22:40:15.697341 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260927 22:40:15.700170 1612 heartbeater.cc:347] Connected to a master server at 127.1.53.126:43947
I20260927 22:40:15.700517 1612 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:15.701278 1612 heartbeater.cc:511] Master 127.1.53.126:43947 requested a full tablet report, sending...
I20260927 22:40:15.702734 1289 ts_manager.cc:194] Registered new tserver with Master: 95a4a4577978412987bc76f30e19c63f (127.1.53.66:40593)
I20260927 22:40:15.703559 1289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:55739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:15.720679 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:43947
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:16.057551 1617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:16.057857 1617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:16.057960 1617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:16.065351 1617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:16.065486 1617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:16.065603 1617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:16.076218 1617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:43947
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:16.077549 1617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:16.078990 1617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:16.093057 1622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:16.093057 1623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:16.093093 1625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:16.094064 1617 server_base.cc:1034] running on GCE node
I20260927 22:40:16.094666 1617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:16.095635 1617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:16.096830 1617 hybrid_clock.cc:648] HybridClock initialized: now 1790548816096793 us; error 38 us; skew 500 ppm
I20260927 22:40:16.097167 1617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:16.100481 1617 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:16.101944 1617 webserver.cc:460] Webserver started at http://127.1.53.67:46371/ using document root <none> and password file <none>
I20260927 22:40:16.102641 1617 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:16.102790 1617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:16.103109 1617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:16.106087 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "498a745dd61a47bc965cb47bf1bfb677"
format_stamp: "Formatted at 2026-09-27 22:40:16 on dist-test-slave-pp80"
server_key: "c36fd27ece039b786cd5dfd18a2b2041"
server_key_iv: "1544fd51a5212acf7341d1f5b9e79491"
server_key_version: "encryptionkey@0"
I20260927 22:40:16.106868 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "498a745dd61a47bc965cb47bf1bfb677"
format_stamp: "Formatted at 2026-09-27 22:40:16 on dist-test-slave-pp80"
server_key: "c36fd27ece039b786cd5dfd18a2b2041"
server_key_iv: "1544fd51a5212acf7341d1f5b9e79491"
server_key_version: "encryptionkey@0"
I20260927 22:40:16.112495 1617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:40:16.116000 1632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:16.117391 1617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:16.117566 1617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "498a745dd61a47bc965cb47bf1bfb677"
format_stamp: "Formatted at 2026-09-27 22:40:16 on dist-test-slave-pp80"
server_key: "c36fd27ece039b786cd5dfd18a2b2041"
server_key_iv: "1544fd51a5212acf7341d1f5b9e79491"
server_key_version: "encryptionkey@0"
I20260927 22:40:16.117758 1617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:16.143566 1617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:16.147989 1617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:16.148401 1617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:16.149379 1617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:16.151162 1617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:16.151266 1617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:16.151409 1617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:16.151481 1617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:16.188491 1617 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:43587
I20260927 22:40:16.188539 1744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:43587 every 8 connection(s)
I20260927 22:40:16.189818 1617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:16.190025 1478 heartbeater.cc:503] Master 127.1.53.126:43947 was elected leader, sending a full tablet report...
I20260927 22:40:16.198609 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 1617
I20260927 22:40:16.198805 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:16.199272 1237 external_mini_cluster.cc:1428] Setting key e945f854e429b15246fff5fba0010a6b
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:16.211627 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.192755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:48443 (local address 127.1.53.126:43947)
0927 22:40:16.192999 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:16.193003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:16.194200 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 22:40:16.195981 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:16.195985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:16.196054 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:16.196148 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:16.197788 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.199257 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.201485 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.202131 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.202232 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.205985 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:16.205990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:16.205993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:16.206024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:16.208710 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.209423 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.209427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.209429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.209559 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.210005 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.210009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.210011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.210289 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:16.210404 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:16.211215 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 22:40:16.211374 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:40:16.212658 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.193314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:43947 (local address 127.1.53.67:48443)
0927 22:40:16.194041 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:16.194080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:16.195714 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:16.196485 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:16.196494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:16.197037 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:16.197611 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:16.197628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.199440 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.199445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:16.201307 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:16.201323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.202372 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.202378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:16.202623 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.203345 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:16.203367 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:16.205741 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:16.208836 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:16.208843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:16.208856 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:16.209301 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:16.209652 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:16.209655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:16.209657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:16.209828 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:16.210425 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:16.210431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:16.210755 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:40:16.211081 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 22:40:16.211471 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":290,"threads_started":1}
I20260927 22:40:16.214267 1745 heartbeater.cc:347] Connected to a master server at 127.1.53.126:43947
I20260927 22:40:16.214610 1745 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:16.215451 1745 heartbeater.cc:511] Master 127.1.53.126:43947 requested a full tablet report, sending...
I20260927 22:40:16.216955 1289 ts_manager.cc:194] Registered new tserver with Master: 498a745dd61a47bc965cb47bf1bfb677 (127.1.53.67:43587)
I20260927 22:40:16.217805 1289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:48443
I20260927 22:40:16.228523 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:16.280063 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.262879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.1.53.65:35903)
0927 22:40:16.263645 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:16.263655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:16.263688 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:40:16.263866 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:16.263875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:16.264136 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:16.264368 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:16.265850 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.266959 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.268414 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.269029 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.269183 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.273370 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:16.273383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:16.273399 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:16.273436 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:16.276962 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.277772 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.277785 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.277807 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.277944 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.278427 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.278432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.278434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.278821 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:16.279142 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:16.279528 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:40:16.279742 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":400,"threads_started":1}
W20260927 22:40:16.281466 1393 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47672
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:16.326086 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.311629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.1.53.65:35903)
0927 22:40:16.311949 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:16.311960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:16.312180 (+ 220us) server_negotiation.cc:407] Connection header received
0927 22:40:16.312438 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:16.312446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:16.312547 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:16.312668 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:16.314077 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.314933 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.316148 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.316616 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.316749 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.320713 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:16.320719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:16.320722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:16.320758 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:16.323586 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.324326 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.324330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.324333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.324468 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.324856 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.324859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.324861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.325124 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:16.325297 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:16.325629 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:40:16.325808 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:40:16.327627 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 1350
I20260927 22:40:16.341578 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.328703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.1.53.65:35903)
0927 22:40:16.328869 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:16.328873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:16.329278 (+ 405us) server_negotiation.cc:407] Connection header received
0927 22:40:16.329386 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:16.329390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:16.329445 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:16.329539 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:16.331367 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.332122 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.333550 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.333990 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.334088 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.336383 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:16.336388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:16.336390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:16.336419 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:16.339002 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.339607 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.339610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.339612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.339738 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.340152 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.340155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.340157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.340450 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:16.340632 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:16.341154 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 22:40:16.341317 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:40:16.342315 1473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:16.429729 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 1350
I20260927 22:40:16.446338 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 1484
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:16.462607 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.447320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.1.53.66:40593)
0927 22:40:16.447850 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:16.447857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:16.447882 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:40:16.447989 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:16.447995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:16.448209 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:16.448408 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:16.449552 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.450621 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.451853 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.452518 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.452618 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.456838 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:16.456849 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:16.456863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:16.456892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:16.459780 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.460512 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.460519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.460521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.460633 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.461000 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.461004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.461006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.461369 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:16.461617 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:16.462000 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 22:40:16.462222 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260927 22:40:16.463937 1607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:16.551453 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 1484
I20260927 22:40:16.574052 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 1617
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:16.591578 1761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:16.575216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.1.53.67:43587)
0927 22:40:16.575851 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:16.575857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:16.575886 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:40:16.575943 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:16.575950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:16.576187 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:16.576382 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:16.577792 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.578985 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.580336 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:16.580922 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:16.581038 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:16.585553 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:16.585566 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:16.585582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:16.585619 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:16.588745 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.589479 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.589487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.589490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.589616 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:16.589972 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:16.589976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:16.589978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:16.590361 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:16.590609 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:16.590926 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:40:16.591173 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":353,"threads_started":1}
I20260927 22:40:16.593128 1740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:16.677667 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 1617
I20260927 22:40:16.694152 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 1257
I20260927 22:40:16.695394 1319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:16.798372 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 1257
2026-09-27T22:40:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4020 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1766](info): set up 2 sockets
Sep 27 22:40:16 dist-test-slave-pp80 krb5kdc[1766](info): commencing operation
krb5kdc: starting...
W20260927 22:40:17.900195 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:18 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:18.080190 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:35531
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:18.402640 1779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:18.402959 1779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:18.403050 1779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:18.410329 1779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:18.410445 1779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:18.410492 1779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:18.410542 1779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:18.410586 1779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:18.420202 1779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:18.421487 1779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:18.422811 1779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:18.435133 1784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:18.435134 1785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:18.435369 1787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:18.435896 1779 server_base.cc:1034] running on GCE node
I20260927 22:40:18.436684 1779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:18.437994 1779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:18.439209 1779 hybrid_clock.cc:648] HybridClock initialized: now 1790548818439143 us; error 53 us; skew 500 ppm
I20260927 22:40:18.439612 1779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:18 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548818, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:18.442889 1779 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:18.444458 1779 webserver.cc:460] Webserver started at http://127.1.53.126:36375/ using document root <none> and password file <none>
I20260927 22:40:18.445147 1779 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:18.445291 1779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:18.445614 1779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:18.448585 1779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "843bf792902045689d0ab7de55663d97"
format_stamp: "Formatted at 2026-09-27 22:40:18 on dist-test-slave-pp80"
server_key: "18c6ec2d63905ae410336def2bb22581"
server_key_iv: "e6d0fc48c5c6a7e1d521b7c9660d4792"
server_key_version: "encryptionkey@0"
I20260927 22:40:18.449319 1779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "843bf792902045689d0ab7de55663d97"
format_stamp: "Formatted at 2026-09-27 22:40:18 on dist-test-slave-pp80"
server_key: "18c6ec2d63905ae410336def2bb22581"
server_key_iv: "e6d0fc48c5c6a7e1d521b7c9660d4792"
server_key_version: "encryptionkey@0"
I20260927 22:40:18.454412 1779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:18.458019 1794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:18.459487 1779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:40:18.459685 1779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "843bf792902045689d0ab7de55663d97"
format_stamp: "Formatted at 2026-09-27 22:40:18 on dist-test-slave-pp80"
server_key: "18c6ec2d63905ae410336def2bb22581"
server_key_iv: "e6d0fc48c5c6a7e1d521b7c9660d4792"
server_key_version: "encryptionkey@0"
I20260927 22:40:18.459888 1779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:18.475751 1779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:18.480161 1779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:18.480460 1779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:18.499254 1779 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:35531
I20260927 22:40:18.499284 1845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:35531 every 8 connection(s)
I20260927 22:40:18.500679 1779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:18.504696 1846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:18.506969 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 1779
I20260927 22:40:18.507192 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:18.507985 1237 external_mini_cluster.cc:1428] Setting key 32ecc60749ba70ce3a1947c501980fab
I20260927 22:40:18.517624 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97: Bootstrap starting.
I20260927 22:40:18.521441 1846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:18.522773 1846 log.cc:824] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:40:18 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:18.525636 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97: No bootstrap required, opened a new log
I20260927 22:40:18.532773 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:18.510301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.1.53.126:35531)
0927 22:40:18.511294 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:18.511307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:18.511352 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:40:18.513293 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:18.513318 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:18.513739 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:18.514112 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:18.515839 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:18.517294 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:18.519001 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:18.519716 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:18.519850 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:18.524758 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:18.524766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:18.524782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:18.524818 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:18.528355 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:18.529088 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:18.529098 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:18.529112 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:18.529275 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:18.529677 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:18.529687 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:18.529691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:18.530265 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:18.530625 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:18.531170 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 22:40:18.531614 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":461,"threads_started":1}
I20260927 22:40:18.533512 1846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843bf792902045689d0ab7de55663d97" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35531 } }
I20260927 22:40:18.533857 1846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:18.533960 1846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843bf792902045689d0ab7de55663d97, State: Initialized, Role: FOLLOWER
I20260927 22:40:18.534646 1846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843bf792902045689d0ab7de55663d97" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35531 } }
I20260927 22:40:18.534929 1846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:18.535059 1846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:18.535228 1846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:18.537701 1846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843bf792902045689d0ab7de55663d97" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35531 } }
I20260927 22:40:18.538110 1846 leader_election.cc:302] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 843bf792902045689d0ab7de55663d97; no voters:
I20260927 22:40:18.538763 1846 leader_election.cc:288] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:18.538893 1851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:18.540012 1851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [term 1 LEADER]: Becoming Leader. State: Replica: 843bf792902045689d0ab7de55663d97, State: Running, Role: LEADER
I20260927 22:40:18.540517 1851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843bf792902045689d0ab7de55663d97" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35531 } }
I20260927 22:40:18.540987 1846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:18.543277 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "843bf792902045689d0ab7de55663d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843bf792902045689d0ab7de55663d97" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35531 } } }
I20260927 22:40:18.543301 1853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 843bf792902045689d0ab7de55663d97. Latest consensus state: current_term: 1 leader_uuid: "843bf792902045689d0ab7de55663d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843bf792902045689d0ab7de55663d97" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35531 } } }
I20260927 22:40:18.543696 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:18.543700 1853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:18.545871 1859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:18.551390 1859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:18.560700 1859 catalog_manager.cc:1347] Generated new cluster ID: ea505d0ea38041b5b06f9e5007d1fcba
I20260927 22:40:18.560819 1859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:18.573468 1859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:18.575431 1859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:18.587131 1859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 843bf792902045689d0ab7de55663d97: Generated new TSK 0
I20260927 22:40:18.588066 1859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:18.640547 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35531
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:18.977010 1872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:18.977299 1872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:18.977470 1872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:18.984891 1872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:18.985006 1872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:18.985188 1872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:18.994943 1872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35531
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:18.996346 1872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:18.997756 1872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:19.012768 1880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:19.012723 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:19.012792 1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:19.013594 1872 server_base.cc:1034] running on GCE node
I20260927 22:40:19.014242 1872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:19.015302 1872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:19.016582 1872 hybrid_clock.cc:648] HybridClock initialized: now 1790548819016526 us; error 54 us; skew 500 ppm
I20260927 22:40:19.016957 1872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:19 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548819, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:19.020639 1872 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:19.022261 1872 webserver.cc:460] Webserver started at http://127.1.53.65:33363/ using document root <none> and password file <none>
I20260927 22:40:19.023028 1872 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:19.023171 1872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:19.023480 1872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:19.026575 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "276157a974d649ef8eab21439a8aefb2"
format_stamp: "Formatted at 2026-09-27 22:40:19 on dist-test-slave-pp80"
server_key: "c31c224028265c376e90ad2a212a5e03"
server_key_iv: "1f78bb9c31d1b5dbb843d9f9214ee0be"
server_key_version: "encryptionkey@0"
I20260927 22:40:19.027370 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "276157a974d649ef8eab21439a8aefb2"
format_stamp: "Formatted at 2026-09-27 22:40:19 on dist-test-slave-pp80"
server_key: "c31c224028265c376e90ad2a212a5e03"
server_key_iv: "1f78bb9c31d1b5dbb843d9f9214ee0be"
server_key_version: "encryptionkey@0"
I20260927 22:40:19.032987 1872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:40:19.036592 1887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:19.037919 1872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:40:19.038137 1872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "276157a974d649ef8eab21439a8aefb2"
format_stamp: "Formatted at 2026-09-27 22:40:19 on dist-test-slave-pp80"
server_key: "c31c224028265c376e90ad2a212a5e03"
server_key_iv: "1f78bb9c31d1b5dbb843d9f9214ee0be"
server_key_version: "encryptionkey@0"
I20260927 22:40:19.038341 1872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:19.063989 1872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:19.068849 1872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:19.069207 1872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:19.070276 1872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:19.071624 1872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:19.071728 1872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:19.071882 1872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:19.071956 1872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:19.106216 1872 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:39371
I20260927 22:40:19.106304 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:39371 every 8 connection(s)
I20260927 22:40:19.107504 1872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:19.109139 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 1872
I20260927 22:40:19.109355 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:19.110183 1237 external_mini_cluster.cc:1428] Setting key e936086a020c761d44ba87000b007429
Sep 27 22:40:19 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548819, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:40:19.131181 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:19.110410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:53671 (local address 127.1.53.126:35531)
0927 22:40:19.116596 (+ 6186us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:19.116601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:19.116617 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:40:19.116663 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:19.116666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:19.116720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:19.116805 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:19.118448 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.119199 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.120906 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.121309 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.121462 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:19.125217 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:19.125221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:19.125225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:19.125255 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:19.128004 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:19.128849 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:19.128856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:19.128859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:19.129003 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:19.129473 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:19.129478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:19.129481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:19.129736 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:19.129888 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:19.130597 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 22:40:19.130812 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6076,"thread_start_us":5980,"threads_started":1}
I20260927 22:40:19.132207 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:19.111096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35531 (local address 127.1.53.65:53671)
0927 22:40:19.112355 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:19.112434 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:19.115788 (+ 3354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:19.117151 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:19.117161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:19.117679 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:19.118263 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:19.118281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.119426 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.119431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:19.120755 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:19.120767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.121553 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.121562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:19.121795 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:19.122502 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:19.122525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:19.124992 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:19.128158 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:19.128168 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:19.128186 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:19.128717 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:19.129103 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:19.129108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:19.129112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:19.129361 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:19.129878 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:19.129886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:19.130230 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 22:40:19.130495 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 22:40:19.131060 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":509,"threads_started":1}
I20260927 22:40:19.133912 2000 heartbeater.cc:347] Connected to a master server at 127.1.53.126:35531
I20260927 22:40:19.134282 2000 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:19.135028 2000 heartbeater.cc:511] Master 127.1.53.126:35531 requested a full tablet report, sending...
I20260927 22:40:19.137373 1811 ts_manager.cc:194] Registered new tserver with Master: 276157a974d649ef8eab21439a8aefb2 (127.1.53.65:39371)
I20260927 22:40:19.139611 1811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:53671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:19.156095 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35531
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:19.478930 2006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:19.479218 2006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:19.479315 2006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:19.486670 2006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:19.486784 2006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:19.486910 2006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:19.497255 2006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35531
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:19.498574 2006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:19.500406 2006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:19.515901 2011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:19.515867 2014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:19.515820 2012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:19.516499 2006 server_base.cc:1034] running on GCE node
I20260927 22:40:19.517124 2006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:19.518095 2006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:19.519320 2006 hybrid_clock.cc:648] HybridClock initialized: now 1790548819519267 us; error 48 us; skew 500 ppm
I20260927 22:40:19.519642 2006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:19 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548819, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:19.522770 2006 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:19.524307 2006 webserver.cc:460] Webserver started at http://127.1.53.66:41835/ using document root <none> and password file <none>
I20260927 22:40:19.524981 2006 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:19.525112 2006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:19.525416 2006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:19.528192 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "dae071a9281f44388c805dc3e0d63c10"
format_stamp: "Formatted at 2026-09-27 22:40:19 on dist-test-slave-pp80"
server_key: "83f4b2de5f1c52e36dc29a2fea436411"
server_key_iv: "db152d2ac95c9cf4738cbbf74e6acde9"
server_key_version: "encryptionkey@0"
I20260927 22:40:19.528996 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "dae071a9281f44388c805dc3e0d63c10"
format_stamp: "Formatted at 2026-09-27 22:40:19 on dist-test-slave-pp80"
server_key: "83f4b2de5f1c52e36dc29a2fea436411"
server_key_iv: "db152d2ac95c9cf4738cbbf74e6acde9"
server_key_version: "encryptionkey@0"
I20260927 22:40:19.534065 2006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:40:19.537566 2021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:19.538902 2006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:19.539163 2006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "dae071a9281f44388c805dc3e0d63c10"
format_stamp: "Formatted at 2026-09-27 22:40:19 on dist-test-slave-pp80"
server_key: "83f4b2de5f1c52e36dc29a2fea436411"
server_key_iv: "db152d2ac95c9cf4738cbbf74e6acde9"
server_key_version: "encryptionkey@0"
I20260927 22:40:19.539371 2006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:19.582376 2006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:19.586855 2006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:19.587158 2006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:19.588228 2006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:19.589586 2006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:19.589656 2006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:19.589736 2006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:19.589781 2006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:19.623499 2006 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:39349
I20260927 22:40:19.623548 2133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:39349 every 8 connection(s)
I20260927 22:40:19.624852 2006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:19.633448 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2006
I20260927 22:40:19.633658 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:19.634181 1237 external_mini_cluster.cc:1428] Setting key a9de98f4753678c947e8b005c0694e3b
Sep 27 22:40:19 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548819, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:19.647384 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:19.627486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:51909 (local address 127.1.53.126:35531)
0927 22:40:19.627701 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:19.627708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:19.629020 (+ 1312us) server_negotiation.cc:407] Connection header received
0927 22:40:19.630803 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:19.630809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:19.630889 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:19.630992 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:19.632417 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.633362 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.635529 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.636235 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.636388 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:19.640446 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:19.640450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:19.640453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:19.640483 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:19.643983 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:19.644845 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:19.644851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:19.644855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:19.645003 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:19.645409 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:19.645416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:19.645422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:19.645773 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:19.645933 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:19.646769 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 22:40:19.647051 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:40:19.648528 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:19.628091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35531 (local address 127.1.53.66:51909)
0927 22:40:19.628844 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:19.628893 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:19.630530 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:19.631187 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:19.631195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:19.631698 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:19.632265 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:19.632282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.633516 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.633522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:19.635397 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:19.635407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:19.636477 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:19.636485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:19.636751 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:19.637463 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:19.637498 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:19.640188 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:19.644126 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:19.644132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:19.644145 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:19.644723 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:19.645109 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:19.645112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:19.645114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:19.645314 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:19.645972 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:19.645978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:19.646309 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 22:40:19.646676 (+ 367us) client_negotiation.cc:240] Negotiation successful
0927 22:40:19.647212 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":337,"threads_started":1}
I20260927 22:40:19.649865 2134 heartbeater.cc:347] Connected to a master server at 127.1.53.126:35531
I20260927 22:40:19.650177 2134 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:19.650866 2134 heartbeater.cc:511] Master 127.1.53.126:35531 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
I20260927 22:40:19.652364 1811 ts_manager.cc:194] Registered new tserver with Master: dae071a9281f44388c805dc3e0d63c10 (127.1.53.66:39349)
I20260927 22:40:19.653239 1811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:51909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:19.676456 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35531
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:20.011142 2139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:20.011421 2139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:20.011587 2139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:20.018805 2139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:20.018929 2139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:20.019143 2139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:20.028955 2139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35531
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:20.030282 2139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:20.031661 2139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:20.045297 2147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:20.045423 2144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:20.045399 2145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:20.046205 2139 server_base.cc:1034] running on GCE node
I20260927 22:40:20.046852 2139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:20.047793 2139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:20.049059 2139 hybrid_clock.cc:648] HybridClock initialized: now 1790548820049026 us; error 66 us; skew 500 ppm
I20260927 22:40:20.049448 2139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:20.052793 2139 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:20.054316 2139 webserver.cc:460] Webserver started at http://127.1.53.67:46217/ using document root <none> and password file <none>
I20260927 22:40:20.055042 2139 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:20.055195 2139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:20.055516 2139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:20.058579 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "46e9f51d368f4359ad2eadae9f771a15"
format_stamp: "Formatted at 2026-09-27 22:40:20 on dist-test-slave-pp80"
server_key: "2eaef2fd4606eab207c684a62130bce3"
server_key_iv: "9c972816df25947a90767eee351596ce"
server_key_version: "encryptionkey@0"
I20260927 22:40:20.059423 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "46e9f51d368f4359ad2eadae9f771a15"
format_stamp: "Formatted at 2026-09-27 22:40:20 on dist-test-slave-pp80"
server_key: "2eaef2fd4606eab207c684a62130bce3"
server_key_iv: "9c972816df25947a90767eee351596ce"
server_key_version: "encryptionkey@0"
I20260927 22:40:20.065129 2139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 22:40:20.069005 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:20.070564 2139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:20.070776 2139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "46e9f51d368f4359ad2eadae9f771a15"
format_stamp: "Formatted at 2026-09-27 22:40:20 on dist-test-slave-pp80"
server_key: "2eaef2fd4606eab207c684a62130bce3"
server_key_iv: "9c972816df25947a90767eee351596ce"
server_key_version: "encryptionkey@0"
I20260927 22:40:20.071086 2139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:20.119360 2139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:20.123838 2139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:20.124145 2139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:20.125201 2139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:20.126564 2139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:20.126668 2139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:20.126813 2139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:20.126885 2139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:20.143390 2000 heartbeater.cc:503] Master 127.1.53.126:35531 was elected leader, sending a full tablet report...
I20260927 22:40:20.162436 2139 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:43253
I20260927 22:40:20.162497 2266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:43253 every 8 connection(s)
I20260927 22:40:20.163846 2139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:20.174067 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2139
I20260927 22:40:20.174291 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:20.174758 1237 external_mini_cluster.cc:1428] Setting key 0484d8d76c2cc0982decae8c0b1a96c9
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:20.184454 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.166524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:41257 (local address 127.1.53.126:35531)
0927 22:40:20.166953 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.166957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.168059 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 22:40:20.169970 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.169976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.170071 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.170188 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.171697 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.172548 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.174125 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.174644 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.174784 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.178823 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.178828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.178831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.178862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.181577 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.182305 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.182309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.182311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.182422 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.182806 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.182809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.182811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.183117 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.183257 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.183941 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.184124 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260927 22:40:20.185560 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.167144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35531 (local address 127.1.53.67:41257)
0927 22:40:20.167837 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:20.167880 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:20.169686 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:20.170430 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:20.170440 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:20.170966 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:20.171524 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.171541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.172718 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.172722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.173979 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.173990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.174890 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.174897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.175169 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.176046 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:20.176070 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:20.178598 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:20.181746 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:20.181754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:20.181768 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:20.182195 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:20.182520 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:20.182523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:20.182525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:20.182705 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:20.183258 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:20.183264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:20.183623 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 22:40:20.183855 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:40:20.184344 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":268,"threads_started":1}
I20260927 22:40:20.187081 2267 heartbeater.cc:347] Connected to a master server at 127.1.53.126:35531
I20260927 22:40:20.187410 2267 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:20.188707 2267 heartbeater.cc:511] Master 127.1.53.126:35531 requested a full tablet report, sending...
I20260927 22:40:20.190331 1811 ts_manager.cc:194] Registered new tserver with Master: 46e9f51d368f4359ad2eadae9f771a15 (127.1.53.67:43253)
I20260927 22:40:20.191148 1811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:41257
I20260927 22:40:20.203107 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:40:20.223446 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.209775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.1.53.126:35531)
0927 22:40:20.209979 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.209984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.210417 (+ 433us) server_negotiation.cc:407] Connection header received
0927 22:40:20.210556 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.210560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.210623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.210717 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.212163 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.213002 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.214389 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.214883 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.215032 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.217485 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.217490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.217494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.217530 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.220719 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.221506 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.221511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.221513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.221629 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.222083 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.222096 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.222099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.222451 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.222657 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.223025 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.223215 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:40:20.230885 1811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:20.238610 1811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:20.267632 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.252490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:39371 (local address 127.0.0.1:33382)
0927 22:40:20.254192 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:20.254237 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:20.254410 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:20.256325 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:20.256337 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:20.256379 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:20.256869 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.256886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.259477 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.259485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.262535 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.262556 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.266348 (+ 3792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.266357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.266557 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.266605 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:40:20.266735 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 22:40:20.266936 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"thread_start_us":13966,"threads_started":1}
I20260927 22:40:20.267694 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.252598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.1.53.65:39371)
0927 22:40:20.253205 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.253218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.254517 (+ 1299us) server_negotiation.cc:407] Connection header received
0927 22:40:20.254849 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.254862 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.255200 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.255417 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:20.257122 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.259254 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.263878 (+ 4624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.265863 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.265984 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.266218 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.266867 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.267125 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":526,"threads_started":1}
I20260927 22:40:20.272948 1935 tablet_service.cc:1467] Processing CreateTablet for tablet dc33046b30634245b955c58294183ebb (DEFAULT_TABLE table=test-table [id=f729bc5ef0b9460da8094b7289f0f33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:20.274520 1935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc33046b30634245b955c58294183ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:20.275372 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.260309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.1.53.66:39349)
0927 22:40:20.262241 (+ 1932us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.262250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.265697 (+ 3447us) server_negotiation.cc:407] Connection header received
0927 22:40:20.265797 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.265805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.266105 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.266306 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:20.267983 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.269970 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.272429 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.273930 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.274062 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.274273 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.274879 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.275053 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":2588,"threads_started":1}
I20260927 22:40:20.275507 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.254964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:39349 (local address 127.0.0.1:58052)
0927 22:40:20.265432 (+ 10468us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:20.265449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:20.265581 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:20.267350 (+ 1769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:20.267354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:20.267377 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:20.267765 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.267779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.270141 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.270146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.272209 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.272224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.274423 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.274429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.274679 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.274719 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:40:20.274814 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:40:20.275075 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10362,"mutex_wait_us":62,"thread_start_us":8308,"threads_started":1}
I20260927 22:40:20.279053 2069 tablet_service.cc:1467] Processing CreateTablet for tablet dc33046b30634245b955c58294183ebb (DEFAULT_TABLE table=test-table [id=f729bc5ef0b9460da8094b7289f0f33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:20.280365 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc33046b30634245b955c58294183ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:20.280725 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.252496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:43253 (local address 127.0.0.1:38494)
0927 22:40:20.263676 (+ 11180us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:20.263702 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:20.263846 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:20.271852 (+ 8006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:20.271857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:20.271884 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:20.272345 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.272359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.274279 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.274286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.276561 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.276576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.279837 (+ 3261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.279855 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.280041 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.280073 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:40:20.280164 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:40:20.280326 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11018,"mutex_wait_us":228,"thread_start_us":979,"threads_started":1}
I20260927 22:40:20.281075 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.252996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.1.53.67:43253)
0927 22:40:20.271250 (+ 18254us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.271258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.271291 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:40:20.271380 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.271388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.271627 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.271825 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:20.272539 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.274107 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.276721 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.278176 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.278275 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.278494 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.280298 (+ 1804us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.280553 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18077,"thread_start_us":17934,"threads_started":1}
I20260927 22:40:20.284862 2202 tablet_service.cc:1467] Processing CreateTablet for tablet dc33046b30634245b955c58294183ebb (DEFAULT_TABLE table=test-table [id=f729bc5ef0b9460da8094b7289f0f33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:20.286300 2202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc33046b30634245b955c58294183ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:20.288393 2286 tablet_bootstrap.cc:492] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: Bootstrap starting.
I20260927 22:40:20.291904 2286 tablet_bootstrap.cc:654] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:20.292975 2286 log.cc:824] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:20.295003 2288 tablet_bootstrap.cc:492] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: Bootstrap starting.
I20260927 22:40:20.296033 2286 tablet_bootstrap.cc:492] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: No bootstrap required, opened a new log
I20260927 22:40:20.296733 2286 ts_tablet_manager.cc:1397] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 22:40:20.298905 2288 tablet_bootstrap.cc:654] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:20.300050 2289 tablet_bootstrap.cc:492] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: Bootstrap starting.
I20260927 22:40:20.300101 2288 log.cc:824] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:20.302842 2289 tablet_bootstrap.cc:654] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:20.304523 2289 log.cc:824] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:20.306263 2288 tablet_bootstrap.cc:492] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: No bootstrap required, opened a new log
I20260927 22:40:20.306535 2288 ts_tablet_manager.cc:1397] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260927 22:40:20.307467 2289 tablet_bootstrap.cc:492] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: No bootstrap required, opened a new log
I20260927 22:40:20.307724 2289 ts_tablet_manager.cc:1397] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:40:20.308110 2286 raft_consensus.cc:348] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.308571 2286 raft_consensus.cc:374] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:20.308683 2286 raft_consensus.cc:729] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276157a974d649ef8eab21439a8aefb2, State: Initialized, Role: FOLLOWER
I20260927 22:40:20.309297 2286 consensus_queue.cc:260] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.311528 2286 ts_tablet_manager.cc:1428] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260927 22:40:20.316323 2289 raft_consensus.cc:348] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.316797 2289 raft_consensus.cc:374] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:20.316934 2289 raft_consensus.cc:729] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9f51d368f4359ad2eadae9f771a15, State: Initialized, Role: FOLLOWER
I20260927 22:40:20.317342 2288 raft_consensus.cc:348] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.317770 2288 raft_consensus.cc:374] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:20.317682 2289 consensus_queue.cc:260] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.317888 2288 raft_consensus.cc:729] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae071a9281f44388c805dc3e0d63c10, State: Initialized, Role: FOLLOWER
I20260927 22:40:20.318428 2288 consensus_queue.cc:260] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.318814 2267 heartbeater.cc:503] Master 127.1.53.126:35531 was elected leader, sending a full tablet report...
I20260927 22:40:20.319465 2134 heartbeater.cc:503] Master 127.1.53.126:35531 was elected leader, sending a full tablet report...
I20260927 22:40:20.319494 2289 ts_tablet_manager.cc:1428] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:40:20.320111 2288 ts_tablet_manager.cc:1428] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260927 22:40:20.360047 2001 tablet.cc:2369] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:40:20.376909 2135 tablet.cc:2369] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:40:20.415673 2268 tablet.cc:2369] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:40:20.440033 2292 raft_consensus.cc:482] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:20.440490 2292 raft_consensus.cc:504] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.442360 2292 leader_election.cc:288] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46e9f51d368f4359ad2eadae9f771a15 (127.1.53.67:43253), dae071a9281f44388c805dc3e0d63c10 (127.1.53.66:39349)
I20260927 22:40:20.452764 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.442818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:46387 (local address 127.1.53.67:43253)
0927 22:40:20.443049 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.443056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.443997 (+ 941us) server_negotiation.cc:407] Connection header received
0927 22:40:20.444262 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.444268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.444366 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.444473 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:20.445288 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.446485 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.449681 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.451303 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.451466 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.451579 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.452362 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.452500 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:40:20.452764 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.443007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:52849 (local address 127.1.53.66:39349)
0927 22:40:20.443194 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.443220 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.443997 (+ 777us) server_negotiation.cc:407] Connection header received
0927 22:40:20.444262 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.444268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.444353 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.444464 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:20.445288 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.446485 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.449681 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.451411 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.451526 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.451611 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.452364 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.452500 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:40:20.453076 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.442776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:39349 (local address 127.1.53.65:52849)
0927 22:40:20.443841 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:20.443861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:20.444028 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:20.444509 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:20.444521 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:20.444566 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:20.445107 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.445127 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.446633 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.446639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.449460 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.449477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.451628 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.451633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.452046 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.452176 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 22:40:20.452337 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:40:20.452557 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":429,"spinlock_wait_cycles":2944,"thread_start_us":664,"threads_started":1}
I20260927 22:40:20.453385 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.442983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:43253 (local address 127.1.53.65:46387)
0927 22:40:20.443883 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:20.443901 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:20.444060 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:20.444509 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:20.444515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:20.444549 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:20.445135 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.445142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.446633 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.446639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.449460 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:20.449477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.451612 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.451621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:20.452045 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.452162 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 22:40:20.452322 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:40:20.452557 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":434,"spinlock_wait_cycles":38528,"thread_start_us":454,"threads_started":1}
I20260927 22:40:20.454487 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33046b30634245b955c58294183ebb" candidate_uuid: "276157a974d649ef8eab21439a8aefb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae071a9281f44388c805dc3e0d63c10" is_pre_election: true
I20260927 22:40:20.454509 2222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33046b30634245b955c58294183ebb" candidate_uuid: "276157a974d649ef8eab21439a8aefb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9f51d368f4359ad2eadae9f771a15" is_pre_election: true
I20260927 22:40:20.455087 2089 raft_consensus.cc:2454] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276157a974d649ef8eab21439a8aefb2 in term 0.
I20260927 22:40:20.455087 2222 raft_consensus.cc:2454] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 276157a974d649ef8eab21439a8aefb2 in term 0.
I20260927 22:40:20.455819 1891 leader_election.cc:302] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 276157a974d649ef8eab21439a8aefb2, 46e9f51d368f4359ad2eadae9f771a15, dae071a9281f44388c805dc3e0d63c10; no voters:
I20260927 22:40:20.456391 2292 raft_consensus.cc:2781] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:20.456569 2292 raft_consensus.cc:482] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:20.456700 2292 raft_consensus.cc:3037] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:20.459301 2292 raft_consensus.cc:504] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.459905 2292 leader_election.cc:288] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [CANDIDATE]: Term 1 election: Requested vote from peers 46e9f51d368f4359ad2eadae9f771a15 (127.1.53.67:43253), dae071a9281f44388c805dc3e0d63c10 (127.1.53.66:39349)
I20260927 22:40:20.460466 2222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33046b30634245b955c58294183ebb" candidate_uuid: "276157a974d649ef8eab21439a8aefb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9f51d368f4359ad2eadae9f771a15"
I20260927 22:40:20.460466 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33046b30634245b955c58294183ebb" candidate_uuid: "276157a974d649ef8eab21439a8aefb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae071a9281f44388c805dc3e0d63c10"
I20260927 22:40:20.460691 2222 raft_consensus.cc:3037] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:20.460691 2089 raft_consensus.cc:3037] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:20.463944 2089 raft_consensus.cc:2454] T dc33046b30634245b955c58294183ebb P dae071a9281f44388c805dc3e0d63c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276157a974d649ef8eab21439a8aefb2 in term 1.
I20260927 22:40:20.463946 2222 raft_consensus.cc:2454] T dc33046b30634245b955c58294183ebb P 46e9f51d368f4359ad2eadae9f771a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 276157a974d649ef8eab21439a8aefb2 in term 1.
I20260927 22:40:20.464429 1891 leader_election.cc:302] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 276157a974d649ef8eab21439a8aefb2, 46e9f51d368f4359ad2eadae9f771a15, dae071a9281f44388c805dc3e0d63c10; no voters:
I20260927 22:40:20.464767 2292 raft_consensus.cc:2781] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:20.465261 2292 raft_consensus.cc:686] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [term 1 LEADER]: Becoming Leader. State: Replica: 276157a974d649ef8eab21439a8aefb2, State: Running, Role: LEADER
I20260927 22:40:20.465757 2292 consensus_queue.cc:237] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } }
I20260927 22:40:20.470810 1811 catalog_manager.cc:5495] T dc33046b30634245b955c58294183ebb P 276157a974d649ef8eab21439a8aefb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 276157a974d649ef8eab21439a8aefb2 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "276157a974d649ef8eab21439a8aefb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276157a974d649ef8eab21439a8aefb2" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 39371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46e9f51d368f4359ad2eadae9f771a15" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 43253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dae071a9281f44388c805dc3e0d63c10" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 39349 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:20.523226 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.506744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.1.53.65:39371)
0927 22:40:20.507023 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.507030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.507408 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:40:20.507578 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.507582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.507642 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.507748 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.510115 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.510914 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.512425 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.512874 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.513019 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.517336 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.517351 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.517368 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.517404 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.520620 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.521311 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.521319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.521321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.521500 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.521863 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.521869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.521872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.522215 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.522447 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.522696 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.522868 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:20.571543 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.556033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.1.53.65:39371)
0927 22:40:20.556316 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.556325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.556702 (+ 377us) server_negotiation.cc:407] Connection header received
0927 22:40:20.556933 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.556937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.557002 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.557111 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.559010 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.559774 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.561182 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.561633 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.561775 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.566018 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.566023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.566027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.566064 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.568871 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.569568 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.569572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.569575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.569736 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.570222 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.570226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.570228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.570511 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.570750 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.570988 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.571246 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:20.616874 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.602200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.1.53.65:39371)
0927 22:40:20.602472 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.602481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.602708 (+ 227us) server_negotiation.cc:407] Connection header received
0927 22:40:20.602921 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.602927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.603026 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.603141 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.604531 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.605265 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.606500 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.606895 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.607022 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.611554 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.611559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.611562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.611593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.614289 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.614948 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.614955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.614959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.615086 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.615447 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.615454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.615457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.615715 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.615969 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.616288 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.616518 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 22:40:20.618607 1915 tablet_service.cc:2908] Rejecting scan request for tablet dc33046b30634245b955c58294183ebb: Uninitialized: clean time has not yet been initialized
I20260927 22:40:20.622138 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 1872
I20260927 22:40:20.636672 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.623170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.1.53.65:39371)
0927 22:40:20.623338 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.623344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.623569 (+ 225us) server_negotiation.cc:407] Connection header received
0927 22:40:20.623703 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.623708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.623797 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.623897 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.625165 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.626275 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.627526 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.628156 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.628292 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.630673 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.630679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.630684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.630718 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.633985 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.634668 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.634673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.634676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.634819 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.635183 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.635187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.635190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.635517 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.635749 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.636038 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.636324 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:40:20.637784 1995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:20.728487 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 1872
I20260927 22:40:20.746017 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2006
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:20.761659 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.747104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.1.53.66:39349)
0927 22:40:20.747275 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.747280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.747299 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:40:20.747413 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.747416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.747494 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.747582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.748895 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.749754 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.751125 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.751535 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.751636 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.755849 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.755868 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.755888 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.755921 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.758857 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.759536 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.759544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.759547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.759679 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.760034 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.760037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.760039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.760542 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.760792 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.761085 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.761266 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:40:20.762360 2129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:20.852905 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2006
I20260927 22:40:20.870913 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2139
Sep 27 22:40:20 dist-test-slave-pp80 krb5kdc[1766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:20.885679 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:20.871718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.1.53.67:43253)
0927 22:40:20.871957 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:20.871966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:20.872001 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:40:20.872068 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:20.872072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:20.872132 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:20.872236 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:20.873487 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.874200 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.875529 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:20.875920 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:20.876065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:20.880325 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:20.880341 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:20.880359 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:20.880400 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:20.882967 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.883610 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.883627 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.883629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.883761 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:20.884138 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:20.884145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:20.884148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:20.884555 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:20.884838 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:20.885130 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:40:20.885347 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:40:20.886369 2262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:20.972905 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2139
I20260927 22:40:20.988865 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 1779
I20260927 22:40:20.990119 1841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:21.098312 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 1779
2026-09-27T22:40:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4298 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:21 dist-test-slave-pp80 krb5kdc[2316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:21 dist-test-slave-pp80 krb5kdc[2316](info): set up 2 sockets
Sep 27 22:40:21 dist-test-slave-pp80 krb5kdc[2316](info): commencing operation
krb5kdc: starting...
W20260927 22:40:22.196484 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:22 dist-test-slave-pp80 krb5kdc[2316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:22.393015 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33991
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:33991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:22.709762 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:22.710132 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:22.710307 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:22.717497 2329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:22.717653 2329 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:40:22.717753 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:22.717870 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:22.717962 2329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:22.718055 2329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:22.727871 2329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:33991
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:22.729211 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:22.730512 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:22.742162 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:22.742192 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:22.742108 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:22.742977 2329 server_base.cc:1034] running on GCE node
I20260927 22:40:22.743696 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:22.745107 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:22.746333 2329 hybrid_clock.cc:648] HybridClock initialized: now 1790548822746256 us; error 52 us; skew 500 ppm
I20260927 22:40:22.746701 2329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:22 dist-test-slave-pp80 krb5kdc[2316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:22.749996 2329 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:22.751482 2329 webserver.cc:460] Webserver started at http://127.1.53.126:35037/ using document root <none> and password file <none>
I20260927 22:40:22.752200 2329 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:22.752357 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:22.752701 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:22.755669 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "22873e8be45d4b18888ff498b6e546f7"
format_stamp: "Formatted at 2026-09-27 22:40:22 on dist-test-slave-pp80"
server_key: "e5cbb9c32e43d3e2a2fcda2f151acb83"
server_key_iv: "c5ba5ffec497330e3511e83b09c69313"
server_key_version: "encryptionkey@0"
I20260927 22:40:22.756551 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "22873e8be45d4b18888ff498b6e546f7"
format_stamp: "Formatted at 2026-09-27 22:40:22 on dist-test-slave-pp80"
server_key: "e5cbb9c32e43d3e2a2fcda2f151acb83"
server_key_iv: "c5ba5ffec497330e3511e83b09c69313"
server_key_version: "encryptionkey@0"
I20260927 22:40:22.761955 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:22.765512 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:22.766949 2329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:22.767144 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "22873e8be45d4b18888ff498b6e546f7"
format_stamp: "Formatted at 2026-09-27 22:40:22 on dist-test-slave-pp80"
server_key: "e5cbb9c32e43d3e2a2fcda2f151acb83"
server_key_iv: "c5ba5ffec497330e3511e83b09c69313"
server_key_version: "encryptionkey@0"
I20260927 22:40:22.767356 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:22.791150 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:22.795835 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:22.796129 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:22.818262 2329 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:33991
I20260927 22:40:22.818272 2405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:33991 every 8 connection(s)
I20260927 22:40:22.819936 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:22.820447 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2329
I20260927 22:40:22.820678 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:22.821521 1237 external_mini_cluster.cc:1428] Setting key cfe193e90469f9c888d6f0053f30e1a9
I20260927 22:40:22.824777 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:40:22 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:22.838567 2406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: Bootstrap starting.
I20260927 22:40:22.842036 2406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:22.843106 2406 log.cc:824] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:22.846194 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:22.823822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.1.53.126:33991)
0927 22:40:22.824710 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:22.824722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:22.824775 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:40:22.826457 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:22.826480 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:22.826880 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:22.827241 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:22.829039 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:22.830411 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:22.832216 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:22.833030 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:22.833172 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:22.838331 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:22.838341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:22.838358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:22.838404 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:22.841676 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:22.842590 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:22.842606 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:22.842614 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:22.842782 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:22.843170 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:22.843175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:22.843179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:22.843996 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:22.844350 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:22.844765 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 22:40:22.845145 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":402,"threads_started":1}
I20260927 22:40:22.846442 2406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: No bootstrap required, opened a new log
I20260927 22:40:22.853698 2406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22873e8be45d4b18888ff498b6e546f7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33991 } }
I20260927 22:40:22.854097 2406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:22.854208 2406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22873e8be45d4b18888ff498b6e546f7, State: Initialized, Role: FOLLOWER
I20260927 22:40:22.854748 2406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22873e8be45d4b18888ff498b6e546f7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33991 } }
I20260927 22:40:22.854978 2406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:22.855120 2406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:22.855288 2406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:22.857465 2406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22873e8be45d4b18888ff498b6e546f7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33991 } }
I20260927 22:40:22.857954 2406 leader_election.cc:302] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22873e8be45d4b18888ff498b6e546f7; no voters:
I20260927 22:40:22.858712 2406 leader_election.cc:288] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:22.858862 2411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:22.859565 2411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [term 1 LEADER]: Becoming Leader. State: Replica: 22873e8be45d4b18888ff498b6e546f7, State: Running, Role: LEADER
I20260927 22:40:22.860153 2411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22873e8be45d4b18888ff498b6e546f7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33991 } }
I20260927 22:40:22.860953 2406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:22.862900 2412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22873e8be45d4b18888ff498b6e546f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22873e8be45d4b18888ff498b6e546f7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33991 } } }
I20260927 22:40:22.863255 2412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:22.865129 2413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22873e8be45d4b18888ff498b6e546f7. Latest consensus state: current_term: 1 leader_uuid: "22873e8be45d4b18888ff498b6e546f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22873e8be45d4b18888ff498b6e546f7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33991 } } }
I20260927 22:40:22.865523 2413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:22.865962 2418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:22.871145 2418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:22.880358 2418 catalog_manager.cc:1347] Generated new cluster ID: 3f1ba0660a6f4f8aa4af518c96bb559c
I20260927 22:40:22.880477 2418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:22.903003 2418 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:22.904440 2418 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:22.915692 2418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: Generated new TSK 0
I20260927 22:40:22.916704 2418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:22.967931 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33991
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:23.312911 2432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:23.313189 2432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:23.313350 2432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:23.320573 2432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:23.320679 2432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:23.320845 2432 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:40:23.320953 2432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:23.330708 2432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:33991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:23.332007 2432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:23.333407 2432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:23.346524 2437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:23.346609 2440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:23.346580 2438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:23.347908 2432 server_base.cc:1034] running on GCE node
I20260927 22:40:23.348569 2432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:23.349509 2432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:23.350775 2432 hybrid_clock.cc:648] HybridClock initialized: now 1790548823350710 us; error 36 us; skew 500 ppm
I20260927 22:40:23.351127 2432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:23 dist-test-slave-pp80 krb5kdc[2316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:23.354384 2432 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:23.355876 2432 webserver.cc:460] Webserver started at http://127.1.53.65:34297/ using document root <none> and password file <none>
I20260927 22:40:23.356666 2432 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:23.356791 2432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:23.357118 2432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:23.359954 2432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "9b799674c02a4d9fb32d0af7dbf67966"
format_stamp: "Formatted at 2026-09-27 22:40:23 on dist-test-slave-pp80"
server_key: "84f6119120646278bb96decf93ce1e7f"
server_key_iv: "fe36de346e25d7064b03ed4f029db335"
server_key_version: "encryptionkey@0"
I20260927 22:40:23.360819 2432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "9b799674c02a4d9fb32d0af7dbf67966"
format_stamp: "Formatted at 2026-09-27 22:40:23 on dist-test-slave-pp80"
server_key: "84f6119120646278bb96decf93ce1e7f"
server_key_iv: "fe36de346e25d7064b03ed4f029db335"
server_key_version: "encryptionkey@0"
I20260927 22:40:23.366374 2432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:23.370114 2447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:23.371524 2432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:40:23.371701 2432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "9b799674c02a4d9fb32d0af7dbf67966"
format_stamp: "Formatted at 2026-09-27 22:40:23 on dist-test-slave-pp80"
server_key: "84f6119120646278bb96decf93ce1e7f"
server_key_iv: "fe36de346e25d7064b03ed4f029db335"
server_key_version: "encryptionkey@0"
I20260927 22:40:23.371917 2432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:23.394379 2432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:23.398764 2432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:23.399058 2432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:23.401872 2432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:23.402004 2432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:23.402153 2432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:23.402231 2432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:40:23 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:23.428378 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.402405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:52009 (local address 127.1.53.126:33991)
0927 22:40:23.403093 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:23.403100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:23.404185 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 22:40:23.406461 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:23.406468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:23.406562 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:23.406681 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:23.409514 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.410240 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.413434 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.414036 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.414184 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.419098 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:23.419105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:23.419109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:23.419153 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:23.423045 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.424177 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.424186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.424192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.424374 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.425743 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.425748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.425752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.426212 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:23.426380 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:23.427586 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0927 22:40:23.427863 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":444,"threads_started":1}
I20260927 22:40:23.430202 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.403119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.1.53.65:52009)
0927 22:40:23.404002 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:23.404057 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:23.406158 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:23.407151 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:23.407168 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:23.408185 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:23.409301 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.409332 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.410401 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.410409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.413227 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.413245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.414387 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.414394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.414778 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.415724 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:23.415798 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:23.418816 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:23.423223 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.423233 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.423259 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.424015 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.424641 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.424649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.424652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.425075 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.426376 (+ 1301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:23.426391 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:23.427057 (+ 666us) client_negotiation.cc:769] Sending connection context
0927 22:40:23.427464 (+ 407us) client_negotiation.cc:240] Negotiation successful
0927 22:40:23.428178 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":379,"threads_started":1}
I20260927 22:40:23.443468 2432 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:42543
I20260927 22:40:23.443562 2567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:42543 every 8 connection(s)
I20260927 22:40:23.445082 2432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:23.447400 2568 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33991
I20260927 22:40:23.447685 2568 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:23.447798 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2432
I20260927 22:40:23.448055 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:23.448552 2568 heartbeater.cc:511] Master 127.1.53.126:33991 requested a full tablet report, sending...
I20260927 22:40:23.448794 1237 external_mini_cluster.cc:1428] Setting key aedc3bbb0a4e485291bcf4e5b9e43455
I20260927 22:40:23.451304 2360 ts_manager.cc:194] Registered new tserver with Master: 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543)
I20260927 22:40:23.453141 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.437683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:51037 (local address 127.1.53.126:33991)
0927 22:40:23.437909 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:23.437918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:23.438778 (+ 860us) server_negotiation.cc:407] Connection header received
0927 22:40:23.438926 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:23.438934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:23.439028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:23.439149 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:23.440725 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.441914 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.443980 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.444632 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.444762 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.447077 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:23.447083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:23.447086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:23.447115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:23.450458 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.451192 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.451199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.451203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.451347 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.451713 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.451717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.451721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.451994 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:23.452150 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:23.452579 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 22:40:23.452781 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:40:23.453204 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.437730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.1.53.65:51037)
0927 22:40:23.438666 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:23.438688 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:23.438805 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:23.439201 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:23.439209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:23.439942 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:23.440533 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.440551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.442060 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.442064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.443855 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.443866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.444817 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.444821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.444944 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.445595 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:23.445625 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:23.446876 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:23.450586 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.450592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.450597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.451075 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.451461 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.451466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.451469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.451601 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.452114 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:23.452119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:23.452347 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 22:40:23.452485 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:40:23.452783 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":624,"threads_started":1}
I20260927 22:40:23.453578 2360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:52009
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:23.491544 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33991
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:23.811359 2572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:23.811677 2572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:23.811774 2572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:23.819139 2572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:23.819239 2572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:23.819379 2572 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:40:23.819490 2572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:23.829339 2572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:33991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:23.830611 2572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:23.831928 2572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:23.845078 2580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:23.845099 2578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:23.845183 2577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:23.849792 2572 server_base.cc:1034] running on GCE node
I20260927 22:40:23.850399 2572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:23.851311 2572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:23.852546 2572 hybrid_clock.cc:648] HybridClock initialized: now 1790548823852486 us; error 44 us; skew 500 ppm
I20260927 22:40:23.852837 2572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:23 dist-test-slave-pp80 krb5kdc[2316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:23.855904 2572 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:23.857539 2572 webserver.cc:460] Webserver started at http://127.1.53.66:34719/ using document root <none> and password file <none>
I20260927 22:40:23.858239 2572 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:23.858364 2572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:23.858742 2572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:23.861703 2572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "863584d275744b4f99bb86dd89989dad"
format_stamp: "Formatted at 2026-09-27 22:40:23 on dist-test-slave-pp80"
server_key: "648cdf564388c970c43b3a5fb7aa2600"
server_key_iv: "0341895d1d9a4016e55595ce53ed6327"
server_key_version: "encryptionkey@0"
I20260927 22:40:23.862509 2572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "863584d275744b4f99bb86dd89989dad"
format_stamp: "Formatted at 2026-09-27 22:40:23 on dist-test-slave-pp80"
server_key: "648cdf564388c970c43b3a5fb7aa2600"
server_key_iv: "0341895d1d9a4016e55595ce53ed6327"
server_key_version: "encryptionkey@0"
I20260927 22:40:23.867556 2572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:40:23.871253 2587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:23.872839 2572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:23.873008 2572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "863584d275744b4f99bb86dd89989dad"
format_stamp: "Formatted at 2026-09-27 22:40:23 on dist-test-slave-pp80"
server_key: "648cdf564388c970c43b3a5fb7aa2600"
server_key_iv: "0341895d1d9a4016e55595ce53ed6327"
server_key_version: "encryptionkey@0"
I20260927 22:40:23.873246 2572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:23.903759 2572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:23.907734 2572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:23.908044 2572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:23.910745 2572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:23.910825 2572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:23.910980 2572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:23.911046 2572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:40:23 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548823, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:23.933626 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.911514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:57939 (local address 127.1.53.126:33991)
0927 22:40:23.911700 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:23.911707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:23.913341 (+ 1634us) server_negotiation.cc:407] Connection header received
0927 22:40:23.916738 (+ 3397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:23.916745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:23.916814 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:23.916898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:23.919697 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.920479 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.922157 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.922596 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.922733 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.927506 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:23.927511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:23.927514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:23.927545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:23.930059 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.930769 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.930776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.930784 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.930893 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.931330 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.931338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.931343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.931615 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:23.931779 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:23.932805 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0927 22:40:23.933083 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:40:23.934732 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.912143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.1.53.66:57939)
0927 22:40:23.913144 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:23.913192 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:23.916323 (+ 3131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:23.917223 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:23.917241 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:23.918410 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:23.919449 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.919485 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.920632 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.920637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.922010 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.922022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.922831 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.922839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.923076 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.923853 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:23.923889 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:23.927285 (+ 3396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:23.930203 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.930212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.930227 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.930641 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.931007 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.931013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.931017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.931222 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.931748 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:23.931757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:23.932277 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 22:40:23.932676 (+ 399us) client_negotiation.cc:240] Negotiation successful
0927 22:40:23.933161 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":418,"threads_started":1}
I20260927 22:40:23.957499 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.941905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:41853 (local address 127.1.53.126:33991)
0927 22:40:23.942113 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:23.942119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:23.943001 (+ 882us) server_negotiation.cc:407] Connection header received
0927 22:40:23.943143 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:23.943146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:23.943207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:23.943289 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:23.944873 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.945814 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.947816 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.948296 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.948451 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.951069 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:23.951077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:23.951082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:23.951121 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:23.954313 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.955246 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.955254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.955260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.955439 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:23.955859 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:23.955866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:23.955871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:23.956389 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:23.956543 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:23.956944 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:40:23.957158 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:40:23.957628 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:23.942009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.1.53.66:41853)
0927 22:40:23.942811 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:23.942836 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:23.943018 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:23.943353 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:23.943361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:23.944102 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:23.944688 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.944710 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.946081 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.946088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.947657 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:23.947668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:23.948505 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:23.948515 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:23.948712 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:23.949426 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:23.949454 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:23.950796 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:23.954452 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.954459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.954466 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.955043 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.955585 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:23.955592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:23.955596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:23.955742 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:23.956531 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:23.956535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:23.956706 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 22:40:23.956854 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:40:23.957178 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
I20260927 22:40:23.969800 2572 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:35413
I20260927 22:40:23.969844 2706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:35413 every 8 connection(s)
I20260927 22:40:23.971011 2572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:23.973196 2707 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33991
I20260927 22:40:23.973431 2707 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:23.974093 2707 heartbeater.cc:511] Master 127.1.53.126:33991 requested a full tablet report, sending...
I20260927 22:40:23.975340 2360 ts_manager.cc:194] Registered new tserver with Master: 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413)
I20260927 22:40:23.976123 2360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:57939
I20260927 22:40:23.979336 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2572
I20260927 22:40:23.979480 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:23.979844 1237 external_mini_cluster.cc:1428] Setting key 4ea6f57c69a2e35aee1110759d800c2a
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:24.019840 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33991
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:24.343961 2711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:24.344223 2711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:24.344389 2711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:24.351498 2711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:24.351606 2711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:24.351687 2711 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:40:24.351768 2711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:24.361500 2711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:33991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:24.362851 2711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:24.364142 2711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:24.377100 2717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:24.377139 2716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:24.377339 2719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:24.378398 2711 server_base.cc:1034] running on GCE node
I20260927 22:40:24.379220 2711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:24.380409 2711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:24.381641 2711 hybrid_clock.cc:648] HybridClock initialized: now 1790548824381571 us; error 51 us; skew 500 ppm
I20260927 22:40:24.381949 2711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:24 dist-test-slave-pp80 krb5kdc[2316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:24.385175 2711 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:24.386569 2711 webserver.cc:460] Webserver started at http://127.1.53.67:37957/ using document root <none> and password file <none>
I20260927 22:40:24.387267 2711 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:24.387396 2711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:24.387715 2711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:24.390647 2711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "0605dac4d1544b89858d268ce0fd7daa"
format_stamp: "Formatted at 2026-09-27 22:40:24 on dist-test-slave-pp80"
server_key: "312f0eaaaff22e616862dc5efff2c0ef"
server_key_iv: "17f7187facc3c6ca82166dcb33a43f71"
server_key_version: "encryptionkey@0"
I20260927 22:40:24.391448 2711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "0605dac4d1544b89858d268ce0fd7daa"
format_stamp: "Formatted at 2026-09-27 22:40:24 on dist-test-slave-pp80"
server_key: "312f0eaaaff22e616862dc5efff2c0ef"
server_key_iv: "17f7187facc3c6ca82166dcb33a43f71"
server_key_version: "encryptionkey@0"
I20260927 22:40:24.396962 2711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:40:24.400564 2726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:24.401911 2711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:40:24.402128 2711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "0605dac4d1544b89858d268ce0fd7daa"
format_stamp: "Formatted at 2026-09-27 22:40:24 on dist-test-slave-pp80"
server_key: "312f0eaaaff22e616862dc5efff2c0ef"
server_key_iv: "17f7187facc3c6ca82166dcb33a43f71"
server_key_version: "encryptionkey@0"
I20260927 22:40:24.402362 2711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:24.422623 2711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:24.426726 2711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:24.427011 2711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:24.429826 2711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:24.429929 2711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:24.430095 2711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:24.430179 2711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:40:24 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:24.456018 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.430305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:53375 (local address 127.1.53.126:33991)
0927 22:40:24.430566 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.430574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.432141 (+ 1567us) server_negotiation.cc:407] Connection header received
0927 22:40:24.435289 (+ 3148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.435296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.435387 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.435482 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:24.438526 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.439855 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.442641 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.443214 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.443310 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.448572 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:24.448578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:24.448585 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:24.448628 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:24.452196 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:24.453246 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:24.453251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:24.453255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:24.453392 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:24.454024 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:24.454030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:24.454035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:24.454367 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:24.454595 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.455608 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.455775 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:40:24.458321 2568 heartbeater.cc:503] Master 127.1.53.126:33991 was elected leader, sending a full tablet report...
I20260927 22:40:24.458860 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.431015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.1.53.67:53375)
0927 22:40:24.431936 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.431997 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.434886 (+ 2889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.436154 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.436169 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.437206 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:24.438278 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.438314 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.440330 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.440341 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.442424 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.442453 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.443689 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.443697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.444004 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.444892 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:24.444920 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:24.448302 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:24.452435 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:24.452445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:24.452470 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:24.453078 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:24.453545 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:24.453553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:24.453560 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:24.453872 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:24.454573 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:24.454584 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:24.455124 (+ 540us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.455501 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.456190 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":361,"threads_started":1}
I20260927 22:40:24.483115 2845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:37325 every 8 connection(s)
I20260927 22:40:24.483180 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.468625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:33497 (local address 127.1.53.126:33991)
0927 22:40:24.469085 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.469092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.469697 (+ 605us) server_negotiation.cc:407] Connection header received
0927 22:40:24.469931 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.469937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.470042 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.470159 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:24.471269 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.472451 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.474585 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.474996 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.475140 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.477329 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:24.477334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:24.477337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:24.477404 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:24.480073 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:24.480905 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:24.480914 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:24.480920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:24.481099 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:24.481510 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:24.481514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:24.481516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:24.481770 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:24.481923 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.482499 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.482758 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260927 22:40:24.483144 2711 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:37325
I20260927 22:40:24.484114 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.468631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.1.53.67:33497)
0927 22:40:24.469562 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.469587 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.469772 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.470193 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.470198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.470707 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:24.471119 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.471130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.472649 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.472660 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.474415 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.474429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.475186 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.475192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.475350 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.476148 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:24.476174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:24.477071 (+ 897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:24.480215 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:24.480221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:24.480225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:24.480768 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:24.481240 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:24.481246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:24.481257 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:24.481400 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:24.481900 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:24.481912 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:24.482156 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.483350 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.483695 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":474,"threads_started":1}
I20260927 22:40:24.484877 2711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:24.487202 2846 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33991
I20260927 22:40:24.487428 2846 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:24.488217 2846 heartbeater.cc:511] Master 127.1.53.126:33991 requested a full tablet report, sending...
I20260927 22:40:24.488380 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2711
I20260927 22:40:24.488560 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:24.488976 1237 external_mini_cluster.cc:1428] Setting key 1b05248085d8044b4248f674d5d8eac5
I20260927 22:40:24.489645 2360 ts_manager.cc:194] Registered new tserver with Master: 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67:37325)
I20260927 22:40:24.490758 2360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:53375
I20260927 22:40:24.492008 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:40:24 dist-test-slave-pp80 krb5kdc[2316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:40:24 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:24.536523 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.522911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.1.53.126:33991)
0927 22:40:24.523130 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.523136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.523327 (+ 191us) server_negotiation.cc:407] Connection header received
0927 22:40:24.523557 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.523562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.523625 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.523731 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:24.525112 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.525874 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.527157 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.527566 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.527711 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.531412 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:24.531417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:24.531420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:24.531452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:24.534012 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:24.534609 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:24.534617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:24.534620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:24.534745 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:24.535115 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:24.535121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:24.535125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:24.535387 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:24.535641 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.535918 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.536139 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:40:24.540584 2361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:24.547224 2361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:24.572337 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.559404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:35413 (local address 127.0.0.1:45958)
0927 22:40:24.561907 (+ 2503us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.561926 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.562041 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.563042 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.563049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.563081 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:24.563603 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.563619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.565793 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.565798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.568771 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.568790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.571078 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.571084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.571282 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.571333 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.571487 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.571684 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2281,"thread_start_us":502,"threads_started":1}
I20260927 22:40:24.572642 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.559646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.1.53.66:35413)
0927 22:40:24.561299 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.561313 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.562157 (+ 844us) server_negotiation.cc:407] Connection header received
0927 22:40:24.562324 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.562333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.562603 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.562800 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:24.563866 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.565648 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.568948 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.570594 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.570721 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.570925 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.572192 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.572372 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":1302,"threads_started":1}
I20260927 22:40:24.576486 2637 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bdf07f6e54f08a8ae2831356d7b80 (DEFAULT_TABLE table=test-table [id=680e76a0378b47c39f1d5e854180f44b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:24.578322 2637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bdf07f6e54f08a8ae2831356d7b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:24.579118 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.559383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37325 (local address 127.0.0.1:50890)
0927 22:40:24.560311 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.560347 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.560571 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.571630 (+ 11059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.571635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.571659 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:24.572071 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.572082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.574793 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.574799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.576412 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.576425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.578361 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.578368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.578543 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.578575 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.578683 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.578802 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":2111,"threads_started":1}
I20260927 22:40:24.579219 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.559970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.1.53.67:37325)
0927 22:40:24.570873 (+ 10903us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.570881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.570916 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:40:24.571034 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.571043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.571321 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.572314 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:24.573182 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.574669 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.576637 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.578178 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.578311 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.578526 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.578730 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.578878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10726,"thread_start_us":10552,"threads_started":1}
I20260927 22:40:24.579622 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.562547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.1.53.65:42543)
0927 22:40:24.570462 (+ 7915us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.570474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.571450 (+ 976us) server_negotiation.cc:407] Connection header received
0927 22:40:24.571545 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.571552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.571786 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.572025 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:24.572709 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.574325 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.576815 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.578220 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.578324 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.578506 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.579220 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.579381 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7711,"thread_start_us":7553,"threads_started":1}
I20260927 22:40:24.580055 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.561700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42543 (local address 127.0.0.1:43110)
0927 22:40:24.571204 (+ 9504us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.571225 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.571339 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.572011 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.572016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.572043 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:24.572513 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.572527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.574487 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.574492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.576661 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.576675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.578687 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.578693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.578850 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.578880 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.578959 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.579085 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9363,"mutex_wait_us":51,"thread_start_us":9907,"threads_started":1}
I20260927 22:40:24.582612 2776 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bdf07f6e54f08a8ae2831356d7b80 (DEFAULT_TABLE table=test-table [id=680e76a0378b47c39f1d5e854180f44b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:24.583910 2776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bdf07f6e54f08a8ae2831356d7b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:24.583626 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bdf07f6e54f08a8ae2831356d7b80 (DEFAULT_TABLE table=test-table [id=680e76a0378b47c39f1d5e854180f44b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:24.584919 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bdf07f6e54f08a8ae2831356d7b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:24.592701 2865 tablet_bootstrap.cc:492] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Bootstrap starting.
I20260927 22:40:24.595541 2866 tablet_bootstrap.cc:492] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Bootstrap starting.
I20260927 22:40:24.596282 2865 tablet_bootstrap.cc:654] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:24.597344 2865 log.cc:824] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:24.599061 2866 tablet_bootstrap.cc:654] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:24.600301 2865 tablet_bootstrap.cc:492] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: No bootstrap required, opened a new log
I20260927 22:40:24.600271 2866 log.cc:824] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:24.600580 2865 ts_tablet_manager.cc:1397] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 22:40:24.600906 2867 tablet_bootstrap.cc:492] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Bootstrap starting.
I20260927 22:40:24.603695 2866 tablet_bootstrap.cc:492] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: No bootstrap required, opened a new log
I20260927 22:40:24.603973 2866 ts_tablet_manager.cc:1397] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:40:24.604709 2867 tablet_bootstrap.cc:654] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:24.605872 2867 log.cc:824] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:24.608879 2867 tablet_bootstrap.cc:492] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: No bootstrap required, opened a new log
I20260927 22:40:24.609145 2867 ts_tablet_manager.cc:1397] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 22:40:24.609421 2865 raft_consensus.cc:348] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.609862 2865 raft_consensus.cc:374] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:24.609968 2865 raft_consensus.cc:729] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863584d275744b4f99bb86dd89989dad, State: Initialized, Role: FOLLOWER
I20260927 22:40:24.610592 2865 consensus_queue.cc:260] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.612941 2707 heartbeater.cc:503] Master 127.1.53.126:33991 was elected leader, sending a full tablet report...
I20260927 22:40:24.613597 2865 ts_tablet_manager.cc:1428] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260927 22:40:24.614378 2866 raft_consensus.cc:348] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.614744 2866 raft_consensus.cc:374] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:24.614857 2866 raft_consensus.cc:729] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0605dac4d1544b89858d268ce0fd7daa, State: Initialized, Role: FOLLOWER
I20260927 22:40:24.615442 2866 consensus_queue.cc:260] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.616473 2846 heartbeater.cc:503] Master 127.1.53.126:33991 was elected leader, sending a full tablet report...
I20260927 22:40:24.617833 2866 ts_tablet_manager.cc:1428] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 22:40:24.621248 2867 raft_consensus.cc:348] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.621609 2867 raft_consensus.cc:374] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:24.621721 2867 raft_consensus.cc:729] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b799674c02a4d9fb32d0af7dbf67966, State: Initialized, Role: FOLLOWER
I20260927 22:40:24.622321 2867 consensus_queue.cc:260] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.623914 2867 ts_tablet_manager.cc:1428] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260927 22:40:24.697841 2569 tablet.cc:2369] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:40:24.723111 2708 tablet.cc:2369] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:40:24.736572 2847 tablet.cc:2369] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:40:24.806070 2872 raft_consensus.cc:482] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:24.806555 2872 raft_consensus.cc:504] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.808341 2872 leader_election.cc:288] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413), 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543)
I20260927 22:40:24.814667 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.808778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:35413 (local address 127.1.53.67:36735)
0927 22:40:24.809026 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.809045 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.809184 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.809510 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.809514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.809539 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:24.809785 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.809799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.810772 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.810777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.812460 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.812474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.814139 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.814143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.814273 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.814297 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.814372 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.814470 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:40:24.814808 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.808794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:36735 (local address 127.1.53.66:35413)
0927 22:40:24.809030 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.809039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.809107 (+ 68us) server_negotiation.cc:407] Connection header received
0927 22:40:24.809337 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.809341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.809396 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.809492 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:24.809934 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.810647 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.812628 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.813985 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.814122 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.814224 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.814428 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.814554 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:40:24.815631 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.809245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42543 (local address 127.1.53.67:40271)
0927 22:40:24.810057 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.810073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.810196 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.810515 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.810519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.810543 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:24.810990 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.811001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.811986 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.811992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.813494 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:24.813505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.815090 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.815096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:24.815250 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.815272 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:40:24.815343 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:40:24.815429 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":520,"threads_started":1}
I20260927 22:40:24.815750 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.809572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:40271 (local address 127.1.53.65:42543)
0927 22:40:24.809803 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.809808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.810156 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:40:24.810317 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.810323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.810397 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.810498 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:24.811164 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.811832 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.813659 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.814897 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.815031 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.815123 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.815403 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.815526 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:40:24.815779 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bdf07f6e54f08a8ae2831356d7b80" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863584d275744b4f99bb86dd89989dad" is_pre_election: true
I20260927 22:40:24.816388 2662 raft_consensus.cc:2454] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 0.
I20260927 22:40:24.816644 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bdf07f6e54f08a8ae2831356d7b80" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b799674c02a4d9fb32d0af7dbf67966" is_pre_election: true
I20260927 22:40:24.817051 2728 leader_election.cc:302] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0605dac4d1544b89858d268ce0fd7daa, 863584d275744b4f99bb86dd89989dad; no voters:
I20260927 22:40:24.817071 2518 raft_consensus.cc:2454] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 0.
I20260927 22:40:24.817414 2872 raft_consensus.cc:2781] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:24.817579 2872 raft_consensus.cc:482] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:24.817690 2872 raft_consensus.cc:3037] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:24.819962 2872 raft_consensus.cc:504] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.820664 2872 leader_election.cc:288] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 election: Requested vote from peers 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413), 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543)
I20260927 22:40:24.821074 2662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bdf07f6e54f08a8ae2831356d7b80" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863584d275744b4f99bb86dd89989dad"
I20260927 22:40:24.821152 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bdf07f6e54f08a8ae2831356d7b80" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b799674c02a4d9fb32d0af7dbf67966"
I20260927 22:40:24.821260 2662 raft_consensus.cc:3037] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:24.821372 2518 raft_consensus.cc:3037] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:24.823457 2518 raft_consensus.cc:2454] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 1.
I20260927 22:40:24.823804 2728 leader_election.cc:302] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0605dac4d1544b89858d268ce0fd7daa, 9b799674c02a4d9fb32d0af7dbf67966; no voters:
I20260927 22:40:24.824084 2872 raft_consensus.cc:2781] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:24.824739 2662 raft_consensus.cc:2454] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 1.
I20260927 22:40:24.824754 2872 raft_consensus.cc:686] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 1 LEADER]: Becoming Leader. State: Replica: 0605dac4d1544b89858d268ce0fd7daa, State: Running, Role: LEADER
I20260927 22:40:24.825536 2872 consensus_queue.cc:237] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } }
I20260927 22:40:24.830183 2360 catalog_manager.cc:5495] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa reported cstate change: term changed from 0 to 1, leader changed from <none> to 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67). New cstate: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } health_report { overall_health: UNKNOWN } } }
I20260927 22:40:24.873284 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.867096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.1.53.67:37325)
0927 22:40:24.867319 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.867328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.867428 (+ 100us) server_negotiation.cc:407] Connection header received
0927 22:40:24.867671 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.867676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.867749 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.867891 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:24.868392 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.869121 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.870394 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:24.870791 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:24.870951 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:24.871980 (+ 1029us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:24.872625 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:24.872772 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:24.872924 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:40:24.873008 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:40:24.881687 2662 raft_consensus.cc:1261] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Refusing update from remote peer 0605dac4d1544b89858d268ce0fd7daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:24.881767 2518 raft_consensus.cc:1261] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Refusing update from remote peer 0605dac4d1544b89858d268ce0fd7daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:24.882617 2875 consensus_queue.cc:1035] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:24.883013 2872 consensus_queue.cc:1035] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:24.893144 2880 mvcc.cc:205] Tried to move back new op lower bound from 7334087986705661952 to 7334087986489098240. Current Snapshot: MvccSnapshot[applied={T|T < 7334087986705661952}]
I20260927 22:40:24.917757 2360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:24.918097 2360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:24.922119 2360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: Sending DeleteTablet for 3 replicas of tablet 0d2bdf07f6e54f08a8ae2831356d7b80
I20260927 22:40:24.923295 2776 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bdf07f6e54f08a8ae2831356d7b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:24 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:50890
I20260927 22:40:24.923774 2637 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bdf07f6e54f08a8ae2831356d7b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:24 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:45958
I20260927 22:40:24.924546 2498 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bdf07f6e54f08a8ae2831356d7b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:24 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:43110
I20260927 22:40:24.924690 2361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:24.925460 2361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:24.925812 2890 tablet_replica.cc:330] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: stopping tablet replica
I20260927 22:40:24.926445 2890 raft_consensus.cc:2229] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:24.927126 2890 raft_consensus.cc:2258] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:24.934557 2636 tablet_service.cc:1467] Processing CreateTablet for tablet d50f7355f0bd4ff7ac9379018a90d85f (DEFAULT_TABLE table=test-table [id=be85b6669c57470cb0b219fde216ed99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:24.935189 2636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50f7355f0bd4ff7ac9379018a90d85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:24.934729 2498 tablet_service.cc:1467] Processing CreateTablet for tablet d50f7355f0bd4ff7ac9379018a90d85f (DEFAULT_TABLE table=test-table [id=be85b6669c57470cb0b219fde216ed99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:24.935632 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50f7355f0bd4ff7ac9379018a90d85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:24.935588 2775 tablet_service.cc:1467] Processing CreateTablet for tablet d50f7355f0bd4ff7ac9379018a90d85f (DEFAULT_TABLE table=test-table [id=be85b6669c57470cb0b219fde216ed99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:24.936146 2775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50f7355f0bd4ff7ac9379018a90d85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:24.942482 2865 tablet_bootstrap.cc:492] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Bootstrap starting.
I20260927 22:40:24.942477 2867 tablet_bootstrap.cc:492] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Bootstrap starting.
I20260927 22:40:24.943902 2889 tablet_replica.cc:330] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: stopping tablet replica
I20260927 22:40:24.944149 2867 tablet_bootstrap.cc:654] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:24.944154 2865 tablet_bootstrap.cc:654] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:24.944459 2889 raft_consensus.cc:2229] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:24.945031 2889 raft_consensus.cc:2258] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:24.947203 2889 ts_tablet_manager.cc:1905] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:24.929567 2890 ts_tablet_manager.cc:1905] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:24.952781 2866 tablet_bootstrap.cc:492] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Bootstrap starting.
I20260927 22:40:24.955015 2866 tablet_bootstrap.cc:654] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:24.955797 2889 ts_tablet_manager.cc:1918] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:24.955977 2889 log.cc:1196] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/0d2bdf07f6e54f08a8ae2831356d7b80
I20260927 22:40:24.956410 2888 tablet_replica.cc:330] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: stopping tablet replica
I20260927 22:40:24.956935 2888 raft_consensus.cc:2229] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:24.957803 2888 raft_consensus.cc:2258] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:24.957963 2867 tablet_bootstrap.cc:492] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: No bootstrap required, opened a new log
I20260927 22:40:24.958148 2867 ts_tablet_manager.cc:1397] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20260927 22:40:24.959074 2867 raft_consensus.cc:348] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.959333 2867 raft_consensus.cc:374] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:24.959448 2867 raft_consensus.cc:729] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b799674c02a4d9fb32d0af7dbf67966, State: Initialized, Role: FOLLOWER
I20260927 22:40:24.959808 2867 consensus_queue.cc:260] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.960096 2888 ts_tablet_manager.cc:1905] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:24.960495 2867 ts_tablet_manager.cc:1428] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260927 22:40:24.968724 2889 ts_tablet_manager.cc:1939] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 863584d275744b4f99bb86dd89989dad: Deleting consensus metadata
I20260927 22:40:24.968766 2890 ts_tablet_manager.cc:1918] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:24.969097 2890 log.cc:1196] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/0d2bdf07f6e54f08a8ae2831356d7b80
I20260927 22:40:24.969740 2890 ts_tablet_manager.cc:1939] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 9b799674c02a4d9fb32d0af7dbf67966: Deleting consensus metadata
I20260927 22:40:24.970925 2346 catalog_manager.cc:4841] TS 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413): tablet 0d2bdf07f6e54f08a8ae2831356d7b80 (table test-table [id=680e76a0378b47c39f1d5e854180f44b]) successfully deleted
I20260927 22:40:24.971777 2888 ts_tablet_manager.cc:1918] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:24.971935 2888 log.cc:1196] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/0d2bdf07f6e54f08a8ae2831356d7b80
I20260927 22:40:24.971954 2346 catalog_manager.cc:4841] TS 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543): tablet 0d2bdf07f6e54f08a8ae2831356d7b80 (table test-table [id=680e76a0378b47c39f1d5e854180f44b]) successfully deleted
I20260927 22:40:24.972486 2888 ts_tablet_manager.cc:1939] T 0d2bdf07f6e54f08a8ae2831356d7b80 P 0605dac4d1544b89858d268ce0fd7daa: Deleting consensus metadata
I20260927 22:40:24.974152 2348 catalog_manager.cc:4841] TS 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67:37325): tablet 0d2bdf07f6e54f08a8ae2831356d7b80 (table test-table [id=680e76a0378b47c39f1d5e854180f44b]) successfully deleted
I20260927 22:40:24.975054 2866 tablet_bootstrap.cc:492] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: No bootstrap required, opened a new log
I20260927 22:40:24.975205 2866 ts_tablet_manager.cc:1397] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20260927 22:40:24.976868 2865 tablet_bootstrap.cc:492] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: No bootstrap required, opened a new log
I20260927 22:40:24.977041 2865 ts_tablet_manager.cc:1397] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.006s
I20260927 22:40:24.977708 2866 raft_consensus.cc:348] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.977946 2866 raft_consensus.cc:374] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:24.978052 2866 raft_consensus.cc:729] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0605dac4d1544b89858d268ce0fd7daa, State: Initialized, Role: FOLLOWER
I20260927 22:40:24.978308 2866 consensus_queue.cc:260] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.978983 2866 ts_tablet_manager.cc:1428] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:40:24.979379 2865 raft_consensus.cc:348] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.979616 2865 raft_consensus.cc:374] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:24.979735 2865 raft_consensus.cc:729] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863584d275744b4f99bb86dd89989dad, State: Initialized, Role: FOLLOWER
I20260927 22:40:24.980031 2865 consensus_queue.cc:260] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.980665 2865 ts_tablet_manager.cc:1428] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:24.993703 2871 raft_consensus.cc:482] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:24.993948 2871 raft_consensus.cc:504] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:24.995858 2871 leader_election.cc:288] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543), 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67:37325)
I20260927 22:40:25.006881 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.997063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:56507 (local address 127.1.53.67:37325)
0927 22:40:24.997233 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.997239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.998812 (+ 1573us) server_negotiation.cc:407] Connection header received
0927 22:40:24.998956 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.998962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.999040 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.999156 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:25.000450 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.001618 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.004616 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.005667 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.005861 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.005980 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.006438 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.006575 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:40:25.007011 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.996684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37325 (local address 127.1.53.66:56507)
0927 22:40:24.998701 (+ 2017us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.998718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.998830 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.999195 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.999200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.999230 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:25.000210 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.000230 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.001774 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.001781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.004377 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.004393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.005922 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.005930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.006158 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.006209 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:40:25.006412 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 22:40:25.006582 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1852,"mutex_wait_us":72,"spinlock_wait_cycles":18816,"thread_start_us":1657,"threads_started":1}
I20260927 22:40:25.007061 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.997055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:53809 (local address 127.1.53.65:42543)
0927 22:40:24.997233 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:24.997239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:24.998425 (+ 1186us) server_negotiation.cc:407] Connection header received
0927 22:40:24.998719 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:24.998725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:24.998817 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:24.998923 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:25.000601 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.001407 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.004549 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.005885 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.006019 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.006120 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.006599 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.006749 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:40:25.007776 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:24.996543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42543 (local address 127.1.53.66:53809)
0927 22:40:24.998288 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:24.998317 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:24.998549 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:24.999330 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:24.999337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:24.999394 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:25.000403 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.000417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.001557 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.001566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.004378 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.004393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.006215 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.006221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.006412 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.006447 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:40:25.007288 (+ 841us) client_negotiation.cc:240] Negotiation successful
0927 22:40:25.007395 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1570,"mutex_wait_us":73,"spinlock_wait_cycles":7936,"thread_start_us":1399,"threads_started":1}
I20260927 22:40:25.008394 2796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50f7355f0bd4ff7ac9379018a90d85f" candidate_uuid: "863584d275744b4f99bb86dd89989dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0605dac4d1544b89858d268ce0fd7daa" is_pre_election: true
I20260927 22:40:25.008525 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50f7355f0bd4ff7ac9379018a90d85f" candidate_uuid: "863584d275744b4f99bb86dd89989dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b799674c02a4d9fb32d0af7dbf67966" is_pre_election: true
I20260927 22:40:25.008769 2518 raft_consensus.cc:2454] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 863584d275744b4f99bb86dd89989dad in term 0.
I20260927 22:40:25.008826 2796 raft_consensus.cc:2454] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 863584d275744b4f99bb86dd89989dad in term 0.
I20260927 22:40:25.009380 2589 leader_election.cc:302] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0605dac4d1544b89858d268ce0fd7daa, 863584d275744b4f99bb86dd89989dad, 9b799674c02a4d9fb32d0af7dbf67966; no voters:
I20260927 22:40:25.010329 2871 raft_consensus.cc:2781] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:25.010464 2871 raft_consensus.cc:482] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:25.010567 2871 raft_consensus.cc:3037] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:25.012921 2871 raft_consensus.cc:504] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.013628 2871 leader_election.cc:288] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [CANDIDATE]: Term 1 election: Requested vote from peers 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543), 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67:37325)
I20260927 22:40:25.014135 2796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50f7355f0bd4ff7ac9379018a90d85f" candidate_uuid: "863584d275744b4f99bb86dd89989dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0605dac4d1544b89858d268ce0fd7daa"
I20260927 22:40:25.014168 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50f7355f0bd4ff7ac9379018a90d85f" candidate_uuid: "863584d275744b4f99bb86dd89989dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b799674c02a4d9fb32d0af7dbf67966"
I20260927 22:40:25.014343 2796 raft_consensus.cc:3037] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:25.014397 2518 raft_consensus.cc:3037] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:25.016660 2518 raft_consensus.cc:2454] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 863584d275744b4f99bb86dd89989dad in term 1.
I20260927 22:40:25.017135 2589 leader_election.cc:302] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863584d275744b4f99bb86dd89989dad, 9b799674c02a4d9fb32d0af7dbf67966; no voters:
I20260927 22:40:25.017444 2871 raft_consensus.cc:2781] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:25.017542 2796 raft_consensus.cc:2454] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 863584d275744b4f99bb86dd89989dad in term 1.
I20260927 22:40:25.017992 2871 raft_consensus.cc:686] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 1 LEADER]: Becoming Leader. State: Replica: 863584d275744b4f99bb86dd89989dad, State: Running, Role: LEADER
I20260927 22:40:25.018435 2871 consensus_queue.cc:237] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.022198 2360 catalog_manager.cc:5495] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad reported cstate change: term changed from 0 to 1, leader changed from <none> to 863584d275744b4f99bb86dd89989dad (127.1.53.66). New cstate: current_term: 1 leader_uuid: "863584d275744b4f99bb86dd89989dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548822, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:25.086086 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.070831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33991 (local address 127.0.0.1:33064)
0927 22:40:25.071640 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:25.071665 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:25.071835 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:25.072158 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:25.072162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:25.072958 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:25.073220 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.073230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.074288 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.074297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.075345 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.075357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.076127 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.076134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.076363 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.077249 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:25.077268 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:25.079931 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:25.083236 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:25.083251 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:25.083260 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:25.083828 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:25.084258 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:25.084263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:25.084266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:25.084434 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:25.084939 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:25.084948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:25.085161 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 22:40:25.085340 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 22:40:25.085640 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":534,"threads_started":1}
I20260927 22:40:25.086102 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.070748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.1.53.126:33991)
0927 22:40:25.071573 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.071580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.071752 (+ 172us) server_negotiation.cc:407] Connection header received
0927 22:40:25.071954 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.071959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.072025 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.072136 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:25.073412 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.074158 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.075489 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.075900 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.076085 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.080152 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:25.080158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:25.080161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:25.080190 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:25.083024 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.083957 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.083961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.083963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.084133 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.084543 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.084547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.084549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.084813 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:25.084923 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.085438 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.085758 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":577,"thread_start_us":397,"threads_started":1}
I20260927 22:40:25.091557 2360 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:33064:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:40:25.093225 2360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:25.100522 2498 tablet_service.cc:1467] Processing CreateTablet for tablet be7d6570c32b428e8474d29e9d0aff84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c5baa6a37204547a7961234d6ded767]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:40:25.100659 2637 tablet_service.cc:1467] Processing CreateTablet for tablet be7d6570c32b428e8474d29e9d0aff84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c5baa6a37204547a7961234d6ded767]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:40:25.101114 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7d6570c32b428e8474d29e9d0aff84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:25.101191 2637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7d6570c32b428e8474d29e9d0aff84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:25.106570 2775 tablet_service.cc:1467] Processing CreateTablet for tablet be7d6570c32b428e8474d29e9d0aff84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c5baa6a37204547a7961234d6ded767]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:40:25.107115 2775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be7d6570c32b428e8474d29e9d0aff84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:25.108422 2865 tablet_bootstrap.cc:492] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: Bootstrap starting.
I20260927 22:40:25.108493 2867 tablet_bootstrap.cc:492] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: Bootstrap starting.
I20260927 22:40:25.110102 2865 tablet_bootstrap.cc:654] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:25.110133 2867 tablet_bootstrap.cc:654] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:25.115540 2866 tablet_bootstrap.cc:492] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: Bootstrap starting.
I20260927 22:40:25.116462 2867 tablet_bootstrap.cc:492] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: No bootstrap required, opened a new log
I20260927 22:40:25.116640 2867 ts_tablet_manager.cc:1397] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 22:40:25.117247 2866 tablet_bootstrap.cc:654] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:25.117223 2867 raft_consensus.cc:348] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.117491 2867 raft_consensus.cc:374] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:25.117583 2867 raft_consensus.cc:729] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b799674c02a4d9fb32d0af7dbf67966, State: Initialized, Role: FOLLOWER
I20260927 22:40:25.117825 2867 consensus_queue.cc:260] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.118391 2867 ts_tablet_manager.cc:1428] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:40:25.118245 2873 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.118618 2873 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.120440 2866 tablet_bootstrap.cc:492] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: No bootstrap required, opened a new log
I20260927 22:40:25.120596 2866 ts_tablet_manager.cc:1397] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 22:40:25.121345 2866 raft_consensus.cc:348] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.121587 2866 raft_consensus.cc:374] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:25.121675 2866 raft_consensus.cc:729] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0605dac4d1544b89858d268ce0fd7daa, State: Initialized, Role: FOLLOWER
I20260927 22:40:25.121910 2866 consensus_queue.cc:260] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.122521 2866 ts_tablet_manager.cc:1428] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:40:25.123404 2865 tablet_bootstrap.cc:492] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: No bootstrap required, opened a new log
I20260927 22:40:25.123561 2865 ts_tablet_manager.cc:1397] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20260927 22:40:25.122424 2872 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.123766 2872 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.124709 2865 raft_consensus.cc:348] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.124948 2865 raft_consensus.cc:374] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:25.125029 2865 raft_consensus.cc:729] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863584d275744b4f99bb86dd89989dad, State: Initialized, Role: FOLLOWER
I20260927 22:40:25.125265 2865 consensus_queue.cc:260] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.125809 2865 ts_tablet_manager.cc:1428] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:40:25.125885 2871 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.126185 2871 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.236776 2872 raft_consensus.cc:482] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:25.237042 2872 raft_consensus.cc:504] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.237900 2872 leader_election.cc:288] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543), 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413)
I20260927 22:40:25.238593 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7d6570c32b428e8474d29e9d0aff84" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b799674c02a4d9fb32d0af7dbf67966" is_pre_election: true
I20260927 22:40:25.238624 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7d6570c32b428e8474d29e9d0aff84" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863584d275744b4f99bb86dd89989dad" is_pre_election: true
I20260927 22:40:25.238853 2518 raft_consensus.cc:2454] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 0.
I20260927 22:40:25.238936 2661 raft_consensus.cc:2454] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 0.
I20260927 22:40:25.239329 2728 leader_election.cc:302] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0605dac4d1544b89858d268ce0fd7daa, 9b799674c02a4d9fb32d0af7dbf67966; no voters:
I20260927 22:40:25.239631 2872 raft_consensus.cc:2781] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:25.239785 2872 raft_consensus.cc:482] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:25.239995 2872 raft_consensus.cc:3037] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:25.242760 2872 raft_consensus.cc:504] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.243324 2872 leader_election.cc:288] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 election: Requested vote from peers 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543), 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413)
I20260927 22:40:25.243852 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7d6570c32b428e8474d29e9d0aff84" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b799674c02a4d9fb32d0af7dbf67966"
I20260927 22:40:25.243929 2661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7d6570c32b428e8474d29e9d0aff84" candidate_uuid: "0605dac4d1544b89858d268ce0fd7daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863584d275744b4f99bb86dd89989dad"
I20260927 22:40:25.244092 2518 raft_consensus.cc:3037] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:25.244123 2661 raft_consensus.cc:3037] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:25.247062 2661 raft_consensus.cc:2454] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 1.
I20260927 22:40:25.247045 2518 raft_consensus.cc:2454] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0605dac4d1544b89858d268ce0fd7daa in term 1.
I20260927 22:40:25.247511 2728 leader_election.cc:302] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0605dac4d1544b89858d268ce0fd7daa, 863584d275744b4f99bb86dd89989dad; no voters:
I20260927 22:40:25.247813 2872 raft_consensus.cc:2781] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:25.248399 2872 raft_consensus.cc:686] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [term 1 LEADER]: Becoming Leader. State: Replica: 0605dac4d1544b89858d268ce0fd7daa, State: Running, Role: LEADER
I20260927 22:40:25.248714 2872 consensus_queue.cc:237] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } }
I20260927 22:40:25.250528 2885 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: TxnStatusTablet state changed. Reason: New leader 0605dac4d1544b89858d268ce0fd7daa. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.250938 2885 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:40:25.251637 2909 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:40:25.251894 2909 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:40:25.253091 2360 catalog_manager.cc:5495] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa reported cstate change: term changed from 0 to 1, leader changed from <none> to 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67). New cstate: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } health_report { overall_health: HEALTHY } } }
I20260927 22:40:25.273829 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.267831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.1.53.67:37325)
0927 22:40:25.268042 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.268049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.268352 (+ 303us) server_negotiation.cc:407] Connection header received
0927 22:40:25.268634 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.268638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.268703 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.268785 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:25.269230 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.269996 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.271958 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.272543 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.272672 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.273193 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.273301 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.273364 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.273547 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.273611 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:40:25.274005 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.267773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37325 (local address 127.0.0.1:50896)
0927 22:40:25.268210 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:25.268266 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:25.268423 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:25.268845 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:25.268850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:25.268864 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:25.269073 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.269084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.270131 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.270137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.271804 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.271820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.272707 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.272713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.272914 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.273007 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.273402 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.273425 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:40:25.273521 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:40:25.273689 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:40:25.277748 2661 raft_consensus.cc:1261] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Refusing update from remote peer 0605dac4d1544b89858d268ce0fd7daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:25.277748 2518 raft_consensus.cc:1261] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Refusing update from remote peer 0605dac4d1544b89858d268ce0fd7daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:25.278398 2885 consensus_queue.cc:1035] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:25.278676 2872 consensus_queue.cc:1035] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:25.285293 2873 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: TxnStatusTablet state changed. Reason: New leader 0605dac4d1544b89858d268ce0fd7daa. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.285694 2873 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.286545 2871 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: TxnStatusTablet state changed. Reason: New leader 0605dac4d1544b89858d268ce0fd7daa. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.286824 2871 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.288062 2878 mvcc.cc:205] Tried to move back new op lower bound from 7334087988330917888 to 7334087988221603840. Current Snapshot: MvccSnapshot[applied={T|T < 7334087988330917888 or (T in {7334087988330917888})}]
I20260927 22:40:25.289067 2885 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.289316 2885 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:40:25.292819 2875 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.293124 2875 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 0605dac4d1544b89858d268ce0fd7daa: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:40:25.293191 2873 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.293404 2873 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 9b799674c02a4d9fb32d0af7dbf67966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.295147 2879 mvcc.cc:205] Tried to move back new op lower bound from 7334087988330917888 to 7334087988221603840. Current Snapshot: MvccSnapshot[applied={T|T < 7334087988330917888}]
I20260927 22:40:25.295459 2871 tablet_replica.cc:439] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0605dac4d1544b89858d268ce0fd7daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 } } peers { permanent_uuid: "863584d275744b4f99bb86dd89989dad" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35413 } } peers { permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 } } }
I20260927 22:40:25.295751 2871 tablet_replica.cc:442] T be7d6570c32b428e8474d29e9d0aff84 P 863584d275744b4f99bb86dd89989dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:25.302454 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.296170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.1.53.66:35413)
0927 22:40:25.296350 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.296356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.296375 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:40:25.296469 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.296475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.296582 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.296748 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:25.297292 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.298441 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.299874 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.300529 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.300660 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.301095 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.301847 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.302020 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.302072 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.302152 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:40:25.320277 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.313135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:48993 (local address 127.1.53.67:37325)
0927 22:40:25.313374 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.313385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.313871 (+ 486us) server_negotiation.cc:407] Connection header received
0927 22:40:25.314048 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.314055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.314159 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.314267 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:25.314904 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.316160 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.318072 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.318764 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.318904 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.319334 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.319475 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.319575 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.319789 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.319902 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:40:25.320295 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.313010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37325 (local address 127.1.53.66:48993)
0927 22:40:25.313756 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:25.313773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:25.313903 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:25.314290 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:25.314293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:25.314303 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:25.314683 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.314699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.316359 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.316364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.317897 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.317911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.318951 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.318957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.319079 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.319139 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.319593 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.319605 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:40:25.319765 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:40:25.319953 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":423,"threads_started":1}
I20260927 22:40:25.331601 2796 raft_consensus.cc:1261] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Refusing update from remote peer 863584d275744b4f99bb86dd89989dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:25.332181 2518 raft_consensus.cc:1261] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Refusing update from remote peer 863584d275744b4f99bb86dd89989dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:25.332871 2896 consensus_queue.cc:1035] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b799674c02a4d9fb32d0af7dbf67966" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:25.333247 2871 consensus_queue.cc:1035] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0605dac4d1544b89858d268ce0fd7daa" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:25.371178 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.364612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:48205 (local address 127.1.53.66:35413)
0927 22:40:25.364828 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.364834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.365516 (+ 682us) server_negotiation.cc:407] Connection header received
0927 22:40:25.365718 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.365724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.365804 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.365911 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:25.366642 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.367655 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.369190 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.369632 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.369765 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.370377 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.370512 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.370613 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.370772 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.370844 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:40:25.371301 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.364685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:35413 (local address 127.1.53.67:48205)
0927 22:40:25.365408 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:25.365429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:25.365592 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:25.366132 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:25.366137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:25.366150 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:25.366497 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.366508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.367769 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.367775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.369044 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:25.369055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.369821 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.369829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:25.370060 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.370129 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:25.370636 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:25.370648 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:40:25.370752 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:40:25.370893 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":343,"threads_started":1}
I20260927 22:40:25.416221 2359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:25.416646 2359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:25.420591 2359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22873e8be45d4b18888ff498b6e546f7: Sending DeleteTablet for 3 replicas of tablet d50f7355f0bd4ff7ac9379018a90d85f
I20260927 22:40:25.421373 2498 tablet_service.cc:1514] Processing DeleteTablet for tablet d50f7355f0bd4ff7ac9379018a90d85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:25 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:43110
I20260927 22:40:25.421576 2637 tablet_service.cc:1514] Processing DeleteTablet for tablet d50f7355f0bd4ff7ac9379018a90d85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:25 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:45958
I20260927 22:40:25.421672 2890 tablet_replica.cc:330] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: stopping tablet replica
I20260927 22:40:25.421875 2889 tablet_replica.cc:330] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: stopping tablet replica
I20260927 22:40:25.421870 2890 raft_consensus.cc:2229] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:25.422076 2889 raft_consensus.cc:2229] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:25.422104 2890 raft_consensus.cc:2258] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:25.422453 2775 tablet_service.cc:1514] Processing DeleteTablet for tablet d50f7355f0bd4ff7ac9379018a90d85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:25 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:50890
I20260927 22:40:25.422573 2889 raft_consensus.cc:2258] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:25.422860 2888 tablet_replica.cc:330] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: stopping tablet replica
I20260927 22:40:25.423065 2888 raft_consensus.cc:2229] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:25.423251 2888 raft_consensus.cc:2258] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:25.423487 2890 ts_tablet_manager.cc:1905] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:25.424875 2888 ts_tablet_manager.cc:1905] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:25.430527 2890 ts_tablet_manager.cc:1918] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:25.430675 2890 log.cc:1196] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/d50f7355f0bd4ff7ac9379018a90d85f
I20260927 22:40:25.431089 2890 ts_tablet_manager.cc:1939] T d50f7355f0bd4ff7ac9379018a90d85f P 9b799674c02a4d9fb32d0af7dbf67966: Deleting consensus metadata
I20260927 22:40:25.432309 2346 catalog_manager.cc:4841] TS 9b799674c02a4d9fb32d0af7dbf67966 (127.1.53.65:42543): tablet d50f7355f0bd4ff7ac9379018a90d85f (table test-table [id=be85b6669c57470cb0b219fde216ed99]) successfully deleted
I20260927 22:40:25.433246 2889 ts_tablet_manager.cc:1905] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:25.433246 2888 ts_tablet_manager.cc:1918] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:25.433413 2888 log.cc:1196] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/d50f7355f0bd4ff7ac9379018a90d85f
I20260927 22:40:25.433880 2888 ts_tablet_manager.cc:1939] T d50f7355f0bd4ff7ac9379018a90d85f P 0605dac4d1544b89858d268ce0fd7daa: Deleting consensus metadata
I20260927 22:40:25.434959 2348 catalog_manager.cc:4841] TS 0605dac4d1544b89858d268ce0fd7daa (127.1.53.67:37325): tablet d50f7355f0bd4ff7ac9379018a90d85f (table test-table [id=be85b6669c57470cb0b219fde216ed99]) successfully deleted
I20260927 22:40:25.442554 2889 ts_tablet_manager.cc:1918] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:25.460471 2889 log.cc:1196] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/d50f7355f0bd4ff7ac9379018a90d85f
I20260927 22:40:25.461266 2889 ts_tablet_manager.cc:1939] T d50f7355f0bd4ff7ac9379018a90d85f P 863584d275744b4f99bb86dd89989dad: Deleting consensus metadata
I20260927 22:40:25.463052 2346 catalog_manager.cc:4841] TS 863584d275744b4f99bb86dd89989dad (127.1.53.66:35413): tablet d50f7355f0bd4ff7ac9379018a90d85f (table test-table [id=be85b6669c57470cb0b219fde216ed99]) successfully deleted
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:25.519968 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.464206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.1.53.65:42543)
0927 22:40:25.464426 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.464433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.475799 (+ 11366us) server_negotiation.cc:407] Connection header received
0927 22:40:25.476021 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.476027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.476128 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.476273 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:25.494468 (+ 18195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.495615 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.497603 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.498229 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.498316 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.512667 (+ 14351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:25.512687 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:25.512709 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:25.512765 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:25.516524 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.517436 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.517446 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.517450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.517598 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.518039 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.518046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.518050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.518478 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:25.518784 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.519312 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.519571 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:40:25.521766 2563 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43120
I20260927 22:40:25.549597 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.525557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.1.53.126:33991)
0927 22:40:25.525723 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.525730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.536762 (+ 11032us) server_negotiation.cc:407] Connection header received
0927 22:40:25.536931 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.536936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.537022 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.537157 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:25.538742 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.539551 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.540941 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.541383 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.541564 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.544045 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:25.544050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:25.544054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:25.544087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:25.546893 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.547550 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.547562 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.547565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.547679 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.548051 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.548055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.548059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.548405 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:25.548641 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.549027 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.549284 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:40:25.550380 2359 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33072
I20260927 22:40:25.552628 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2432
I20260927 22:40:25.567714 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.555241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.1.53.65:42543)
0927 22:40:25.555419 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.555425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.555445 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:40:25.555503 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.555507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.555591 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.555686 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:25.556862 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.557616 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.558803 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.559214 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.559362 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.562106 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:25.562117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:25.562121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:25.562157 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:25.565009 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.565718 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.565740 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.565743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.565914 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.566273 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.566279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.566282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.566542 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:25.566809 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.567108 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.567325 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 22:40:25.568399 2563 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43136
W20260927 22:40:25.568956 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2432: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:25.569144 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2432
I20260927 22:40:25.588174 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2572
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:25.604403 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.589596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.1.53.66:35413)
0927 22:40:25.589817 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.589822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.589838 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:40:25.589893 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.589897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.589960 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.590084 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:25.591584 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.592410 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.593673 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.594143 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.594276 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.598439 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:25.598452 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:25.598469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:25.598501 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:25.601541 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.602193 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.602201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.602203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.602345 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.602726 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.602733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.602736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.603094 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:25.603421 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.603775 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.604009 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 22:40:25.605185 2702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45976
W20260927 22:40:25.605937 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:25.606127 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2572
I20260927 22:40:25.630903 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2711
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:25.648536 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:25.631722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.1.53.67:37325)
0927 22:40:25.631914 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:25.631918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:25.632019 (+ 101us) server_negotiation.cc:407] Connection header received
0927 22:40:25.632184 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:25.632188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:25.632324 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:25.632408 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:25.634030 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.634784 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.636622 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:25.637124 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:25.637228 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:25.642313 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:25.642329 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:25.642351 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:25.642394 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:25.645729 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.646476 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.646485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.646487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.646629 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:25.647010 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:25.647017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:25.647026 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:25.647400 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:25.647727 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:25.647954 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:40:25.648200 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:40:25.649358 2841 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50910
W20260927 22:40:25.650079 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2711: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:25.650235 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2711
I20260927 22:40:25.673249 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2329
I20260927 22:40:25.674554 2401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:25.789222 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2329
2026-09-27T22:40:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4691 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2931](info): set up 2 sockets
Sep 27 22:40:25 dist-test-slave-pp80 krb5kdc[2931](info): commencing operation
krb5kdc: starting...
W20260927 22:40:26.884326 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:27 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:27.078365 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:40867
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:27.401309 2944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:27.401611 2944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:27.401762 2944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:27.409157 2944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:27.409261 2944 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:40:27.409338 2944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:27.409453 2944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:27.409526 2944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:27.409634 2944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:27.419629 2944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:40867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:27.421011 2944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:27.422380 2944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:27.434574 2950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:27.434499 2952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:27.434551 2949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:27.437747 2944 server_base.cc:1034] running on GCE node
I20260927 22:40:27.438614 2944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:27.440115 2944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:27.441342 2944 hybrid_clock.cc:648] HybridClock initialized: now 1790548827441267 us; error 48 us; skew 500 ppm
I20260927 22:40:27.441711 2944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:27 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:27.445035 2944 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:27.446553 2944 webserver.cc:460] Webserver started at http://127.1.53.126:46511/ using document root <none> and password file <none>
I20260927 22:40:27.447271 2944 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:27.447381 2944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:27.447708 2944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:27.450641 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "82dedba1cb2a4d569bce05d0850b1e86"
format_stamp: "Formatted at 2026-09-27 22:40:27 on dist-test-slave-pp80"
server_key: "4d2e685c1c1693c8b1d82657ebaa7de6"
server_key_iv: "eb1813bc9d4378d59453bf4fb2c6c1cb"
server_key_version: "encryptionkey@0"
I20260927 22:40:27.451401 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "82dedba1cb2a4d569bce05d0850b1e86"
format_stamp: "Formatted at 2026-09-27 22:40:27 on dist-test-slave-pp80"
server_key: "4d2e685c1c1693c8b1d82657ebaa7de6"
server_key_iv: "eb1813bc9d4378d59453bf4fb2c6c1cb"
server_key_version: "encryptionkey@0"
I20260927 22:40:27.456773 2944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 22:40:27.460445 2959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:27.461814 2944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:27.461984 2944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "82dedba1cb2a4d569bce05d0850b1e86"
format_stamp: "Formatted at 2026-09-27 22:40:27 on dist-test-slave-pp80"
server_key: "4d2e685c1c1693c8b1d82657ebaa7de6"
server_key_iv: "eb1813bc9d4378d59453bf4fb2c6c1cb"
server_key_version: "encryptionkey@0"
I20260927 22:40:27.462203 2944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:27.483307 2944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:27.487717 2944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:27.487978 2944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:27.508080 2944 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:40867
I20260927 22:40:27.508098 3020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:40867 every 8 connection(s)
I20260927 22:40:27.509487 2944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:27.513314 3021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:27.516824 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 2944
I20260927 22:40:27.517100 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:27.517781 1237 external_mini_cluster.cc:1428] Setting key 67044276363cb9e29bf20c7dc18057cc
I20260927 22:40:27.523983 3021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Bootstrap starting.
I20260927 22:40:27.527534 3021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:27.528728 3021 log.cc:824] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:27.531466 3021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: No bootstrap required, opened a new log
Sep 27 22:40:27 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:27.542244 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:27.519780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.1.53.126:40867)
0927 22:40:27.520630 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:27.520643 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:27.520692 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:40:27.522717 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:27.522752 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:27.523152 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:27.523498 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:27.525209 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:27.526676 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:27.528452 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:27.529276 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:27.529504 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:27.533626 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:27.533636 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:27.533657 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:27.533714 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:27.537788 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:27.538458 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:27.538466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:27.538474 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:27.538677 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:27.539031 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:27.539035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:27.539039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:27.539723 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:27.540029 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:27.540457 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 22:40:27.540897 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":413,"threads_started":1}
I20260927 22:40:27.542043 3021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dedba1cb2a4d569bce05d0850b1e86" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40867 } }
I20260927 22:40:27.542469 3021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:27.542574 3021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dedba1cb2a4d569bce05d0850b1e86, State: Initialized, Role: FOLLOWER
I20260927 22:40:27.543197 3021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dedba1cb2a4d569bce05d0850b1e86" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40867 } }
I20260927 22:40:27.543470 3021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:27.543608 3021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:27.543821 3021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:27.546295 3021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dedba1cb2a4d569bce05d0850b1e86" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40867 } }
I20260927 22:40:27.546779 3021 leader_election.cc:302] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82dedba1cb2a4d569bce05d0850b1e86; no voters:
I20260927 22:40:27.547327 3021 leader_election.cc:288] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:27.547494 3026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:27.548372 3026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [term 1 LEADER]: Becoming Leader. State: Replica: 82dedba1cb2a4d569bce05d0850b1e86, State: Running, Role: LEADER
I20260927 22:40:27.548921 3026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dedba1cb2a4d569bce05d0850b1e86" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40867 } }
I20260927 22:40:27.549602 3021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:27.552258 3028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82dedba1cb2a4d569bce05d0850b1e86. Latest consensus state: current_term: 1 leader_uuid: "82dedba1cb2a4d569bce05d0850b1e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dedba1cb2a4d569bce05d0850b1e86" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40867 } } }
I20260927 22:40:27.552594 3028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:27.553443 3027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82dedba1cb2a4d569bce05d0850b1e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dedba1cb2a4d569bce05d0850b1e86" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40867 } } }
I20260927 22:40:27.553774 3027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:27.557906 3034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:27.564972 3034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:27.574330 3034 catalog_manager.cc:1347] Generated new cluster ID: 5d177449eccf43e196b29aa1ef865f7b
I20260927 22:40:27.574447 3034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:27.602553 3034 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:27.603937 3034 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:27.611580 3034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Generated new TSK 0
I20260927 22:40:27.612457 3034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:27.668075 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:40867
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:27.990315 3047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:27.990579 3047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:27.990741 3047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:27.997792 3047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:27.997906 3047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:27.998064 3047 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:40:27.998142 3047 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:40:27.998272 3047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:28.008472 3047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:40867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:28.009840 3047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:28.011168 3047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:28.024871 3052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:28.024901 3053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:28.024919 3055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:28.025774 3047 server_base.cc:1034] running on GCE node
I20260927 22:40:28.026422 3047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:28.027407 3047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:28.028601 3047 hybrid_clock.cc:648] HybridClock initialized: now 1790548828028549 us; error 42 us; skew 500 ppm
I20260927 22:40:28.028932 3047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:28 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:28.032357 3047 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:28.033821 3047 webserver.cc:460] Webserver started at http://127.1.53.65:35261/ using document root <none> and password file <none>
I20260927 22:40:28.034571 3047 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:28.034674 3047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:28.035024 3047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:28.037968 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "0f9802a3edc34b489ec509361e74724d"
format_stamp: "Formatted at 2026-09-27 22:40:28 on dist-test-slave-pp80"
server_key: "e1902dcad06dab772ec63f93e5f5eea5"
server_key_iv: "e3cdce780daaa4237ebe64c16b29332e"
server_key_version: "encryptionkey@0"
I20260927 22:40:28.038723 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "0f9802a3edc34b489ec509361e74724d"
format_stamp: "Formatted at 2026-09-27 22:40:28 on dist-test-slave-pp80"
server_key: "e1902dcad06dab772ec63f93e5f5eea5"
server_key_iv: "e3cdce780daaa4237ebe64c16b29332e"
server_key_version: "encryptionkey@0"
I20260927 22:40:28.043996 3047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:40:28.047453 3062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:28.048842 3047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:28.049008 3047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "0f9802a3edc34b489ec509361e74724d"
format_stamp: "Formatted at 2026-09-27 22:40:28 on dist-test-slave-pp80"
server_key: "e1902dcad06dab772ec63f93e5f5eea5"
server_key_iv: "e3cdce780daaa4237ebe64c16b29332e"
server_key_version: "encryptionkey@0"
I20260927 22:40:28.049218 3047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:28.064899 3047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:28.069572 3047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:28.069895 3047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:28.072649 3047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:28.072777 3047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:28.072932 3047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:28.073032 3047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:40:28 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:28.095651 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.073255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:37277 (local address 127.1.53.126:40867)
0927 22:40:28.074031 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:28.074038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:28.075072 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 22:40:28.077361 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:28.077368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:28.077458 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:28.077613 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:28.079018 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.080503 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.082981 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.083552 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.083681 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.088659 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:28.088667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:28.088674 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:28.088715 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:28.092218 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.093150 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.093154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.093156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.093259 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.093734 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.093739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.093745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.094007 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:28.094168 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:28.095075 (+ 907us) server_negotiation.cc:299] Negotiation successful
0927 22:40:28.095291 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":521,"threads_started":1}
I20260927 22:40:28.097308 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.073973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.1.53.65:37277)
0927 22:40:28.074887 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:28.074937 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:28.077027 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:28.077811 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:28.077819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:28.078291 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:28.078845 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.078860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.080663 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.080670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.082784 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.082802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.083837 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.083847 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.084087 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.085008 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:28.085029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:28.088347 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:28.092394 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.092404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.092421 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.092971 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.093371 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.093376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.093380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.093609 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.094141 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:28.094153 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:28.094551 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 22:40:28.094986 (+ 435us) client_negotiation.cc:240] Negotiation successful
0927 22:40:28.095527 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":389,"threads_started":1}
I20260927 22:40:28.117620 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.104476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.1.53.65:37287)
0927 22:40:28.105125 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:28.105149 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:28.105351 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:28.105680 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:28.105687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:28.106117 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:28.106377 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.106385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.107672 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.107677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.108935 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.108947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.109675 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.109681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.109850 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.110877 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:28.110901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:28.111698 (+ 797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:28.114884 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.114890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.114894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.115314 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.115672 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.115676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.115680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.115885 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.116554 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:28.116559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:28.116821 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 22:40:28.117017 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 22:40:28.117304 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":355,"threads_started":1}
I20260927 22:40:28.118052 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.104421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:37287 (local address 127.1.53.126:40867)
0927 22:40:28.104590 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:28.104598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:28.105233 (+ 635us) server_negotiation.cc:407] Connection header received
0927 22:40:28.105478 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:28.105483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:28.105559 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:28.105656 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:28.106508 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.107538 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.109097 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.109492 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.109632 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.111884 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:28.111892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:28.111906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:28.111933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:28.114758 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.115435 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.115440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.115444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.115566 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.116041 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.116047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.116051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.116421 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:28.116568 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:28.117191 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 22:40:28.117527 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:40:28.125525 3047 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:38729
I20260927 22:40:28.125572 3182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:38729 every 8 connection(s)
I20260927 22:40:28.126902 3047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:28.127413 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 3047
I20260927 22:40:28.127614 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:28.128342 1237 external_mini_cluster.cc:1428] Setting key cbba07e0fa47815d04ec15b9cfdfc48f
I20260927 22:40:28.129346 3183 heartbeater.cc:347] Connected to a master server at 127.1.53.126:40867
I20260927 22:40:28.129585 3183 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:28.130249 3183 heartbeater.cc:511] Master 127.1.53.126:40867 requested a full tablet report, sending...
I20260927 22:40:28.132179 2976 ts_manager.cc:194] Registered new tserver with Master: 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729)
I20260927 22:40:28.133725 2976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:37277
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:28.172536 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:40867
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:28.496553 3187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:28.496866 3187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:28.497040 3187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:28.504441 3187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:28.504594 3187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:28.504738 3187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:40:28.504825 3187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:40:28.504959 3187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:28.519480 3187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:40867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:28.521096 3187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:28.522742 3187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:28.537541 3195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:28.537510 3193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:28.537590 3192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:28.538293 3187 server_base.cc:1034] running on GCE node
I20260927 22:40:28.538923 3187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:28.539878 3187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:28.541170 3187 hybrid_clock.cc:648] HybridClock initialized: now 1790548828541125 us; error 60 us; skew 500 ppm
I20260927 22:40:28.541564 3187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:28 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:28.545029 3187 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:28.546515 3187 webserver.cc:460] Webserver started at http://127.1.53.66:35479/ using document root <none> and password file <none>
I20260927 22:40:28.547212 3187 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:28.547353 3187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:28.547658 3187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:28.550788 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "3b80662cc6384b1692d189fce3092f9c"
format_stamp: "Formatted at 2026-09-27 22:40:28 on dist-test-slave-pp80"
server_key: "56d2743080f63e448052ace010a3a9f8"
server_key_iv: "170f14639ea00bf33d6a0b2be84b7e95"
server_key_version: "encryptionkey@0"
I20260927 22:40:28.551594 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "3b80662cc6384b1692d189fce3092f9c"
format_stamp: "Formatted at 2026-09-27 22:40:28 on dist-test-slave-pp80"
server_key: "56d2743080f63e448052ace010a3a9f8"
server_key_iv: "170f14639ea00bf33d6a0b2be84b7e95"
server_key_version: "encryptionkey@0"
I20260927 22:40:28.557158 3187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:40:28.561386 3202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:28.562822 3187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260927 22:40:28.563041 3187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "3b80662cc6384b1692d189fce3092f9c"
format_stamp: "Formatted at 2026-09-27 22:40:28 on dist-test-slave-pp80"
server_key: "56d2743080f63e448052ace010a3a9f8"
server_key_iv: "170f14639ea00bf33d6a0b2be84b7e95"
server_key_version: "encryptionkey@0"
I20260927 22:40:28.563265 3187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:28.582465 3187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:28.586789 3187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:28.587086 3187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:28.589927 3187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:28.590073 3187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:28.590196 3187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:28.590288 3187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:40:28 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:28.619069 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.591165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.1.53.66:49637)
0927 22:40:28.592099 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:28.592149 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:28.595020 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:28.596348 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:28.596363 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:28.597353 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:28.598440 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.598472 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.600293 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.600299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.602518 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.602534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.603860 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.603866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.604078 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.604988 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:28.605016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:28.608802 (+ 3786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:28.613249 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.613261 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.613291 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.614035 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.614472 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.614488 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.614491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.614988 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.615779 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:28.615788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:28.616450 (+ 662us) client_negotiation.cc:769] Sending connection context
0927 22:40:28.616827 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 22:40:28.617583 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":389,"threads_started":1}
I20260927 22:40:28.620229 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.590546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:49637 (local address 127.1.53.126:40867)
0927 22:40:28.590752 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:28.590758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:28.592312 (+ 1554us) server_negotiation.cc:407] Connection header received
0927 22:40:28.595425 (+ 3113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:28.595434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:28.595580 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:28.595674 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:28.598688 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.600107 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.602783 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.603603 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.603762 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.609097 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:28.609106 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:28.609114 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:28.609171 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:28.613028 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.614189 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.614195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.614199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.614336 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.615146 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.615151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.615156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.615608 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:28.615781 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:28.619664 (+ 3883us) server_negotiation.cc:299] Negotiation successful
0927 22:40:28.619934 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:40:28.641247 3187 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:33125
I20260927 22:40:28.641330 3321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:33125 every 8 connection(s)
I20260927 22:40:28.641650 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.627521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:49239 (local address 127.1.53.126:40867)
0927 22:40:28.627725 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:28.627734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:28.627869 (+ 135us) server_negotiation.cc:407] Connection header received
0927 22:40:28.628022 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:28.628030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:28.628127 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:28.628319 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:28.629335 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.630081 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.631561 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.631996 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.632162 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.634994 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:28.635002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:28.635012 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:28.635043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:28.638235 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.639220 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.639225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.639229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.639382 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:28.639809 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:28.639814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:28.639818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:28.640353 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:28.640518 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:28.641095 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 22:40:28.641331 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:40:28.641866 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:28.626998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.1.53.66:49239)
0927 22:40:28.627754 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:28.627782 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:28.627899 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:28.628343 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:28.628346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:28.628783 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:28.629165 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.629175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.630236 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.630250 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.631409 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:28.631421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:28.632206 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:28.632213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:28.632474 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:28.633557 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:28.633587 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:28.634772 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:28.638445 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.638453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.638463 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.639082 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.639518 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:28.639522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:28.639524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:28.639686 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:28.640493 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:28.640499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:28.640748 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 22:40:28.640944 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 22:40:28.641376 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":463,"threads_started":1}
I20260927 22:40:28.642897 3187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:28.645104 3322 heartbeater.cc:347] Connected to a master server at 127.1.53.126:40867
I20260927 22:40:28.645324 3322 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:28.646093 3322 heartbeater.cc:511] Master 127.1.53.126:40867 requested a full tablet report, sending...
I20260927 22:40:28.647326 2976 ts_manager.cc:194] Registered new tserver with Master: 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125)
I20260927 22:40:28.648104 2976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:49637
I20260927 22:40:28.651120 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 3187
I20260927 22:40:28.651250 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:28.651625 1237 external_mini_cluster.cc:1428] Setting key 7cf85e1aaadc146eaa7886ca3a8983d2
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:28.695850 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:40867
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:29.018764 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:29.019060 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:29.019161 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:29.026466 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:29.026577 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:29.026662 3326 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:40:29.026717 3326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:40:29.026777 3326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:29.036736 3326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:40867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:29.038057 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:29.039353 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:29.052672 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:29.052752 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:29.052712 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:29.053678 3326 server_base.cc:1034] running on GCE node
I20260927 22:40:29.054386 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:29.055395 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:29.056638 3326 hybrid_clock.cc:648] HybridClock initialized: now 1790548829056560 us; error 67 us; skew 500 ppm
I20260927 22:40:29.056989 3326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:29 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:29.060806 3326 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:29.062377 3326 webserver.cc:460] Webserver started at http://127.1.53.67:43873/ using document root <none> and password file <none>
I20260927 22:40:29.063095 3326 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:29.063202 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:29.063488 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:29.067059 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "1897cbf984014c6599441d06952a1c72"
format_stamp: "Formatted at 2026-09-27 22:40:29 on dist-test-slave-pp80"
server_key: "e6f02b008419ba0b4b59c40db15f1f37"
server_key_iv: "ec43aa95a668bba0d7813bc909f5a0f3"
server_key_version: "encryptionkey@0"
I20260927 22:40:29.067874 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "1897cbf984014c6599441d06952a1c72"
format_stamp: "Formatted at 2026-09-27 22:40:29 on dist-test-slave-pp80"
server_key: "e6f02b008419ba0b4b59c40db15f1f37"
server_key_iv: "ec43aa95a668bba0d7813bc909f5a0f3"
server_key_version: "encryptionkey@0"
I20260927 22:40:29.073630 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:40:29.077383 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:29.078912 3326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:29.079125 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "1897cbf984014c6599441d06952a1c72"
format_stamp: "Formatted at 2026-09-27 22:40:29 on dist-test-slave-pp80"
server_key: "e6f02b008419ba0b4b59c40db15f1f37"
server_key_iv: "ec43aa95a668bba0d7813bc909f5a0f3"
server_key_version: "encryptionkey@0"
I20260927 22:40:29.079301 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:29.097785 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:29.102245 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:29.102531 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:29.105310 3326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:29.105456 3326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:29.105582 3326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:29.105700 3326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:40:29 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:29.130501 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.105885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:36667 (local address 127.1.53.126:40867)
0927 22:40:29.106135 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.106141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.107553 (+ 1412us) server_negotiation.cc:407] Connection header received
0927 22:40:29.109663 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.109669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.109750 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.109873 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:29.112117 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.113236 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.115774 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.116507 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.116677 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.122202 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:29.122211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:29.122218 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:29.122274 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:29.126160 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.127450 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.127455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.127459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.127619 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.128289 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.128294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.128297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.128658 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:29.128801 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.129902 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.130147 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 22:40:29.132375 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.106491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.1.53.67:36667)
0927 22:40:29.107350 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.107411 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.109335 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.110125 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.110133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.110890 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:29.111854 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.111882 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.113404 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.113411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.115561 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.115598 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.116841 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.116848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.117153 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.118113 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:29.118157 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:29.121865 (+ 3708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:29.126426 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:29.126445 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:29.126470 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:29.127269 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:29.127775 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:29.127782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:29.127789 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:29.128118 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:29.128818 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:29.128828 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:29.129393 (+ 565us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.129783 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.130483 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":405,"threads_started":1}
I20260927 22:40:29.137213 3183 heartbeater.cc:503] Master 127.1.53.126:40867 was elected leader, sending a full tablet report...
I20260927 22:40:29.152129 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.138827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:52873 (local address 127.1.53.126:40867)
0927 22:40:29.139040 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.139049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.139130 (+ 81us) server_negotiation.cc:407] Connection header received
0927 22:40:29.139342 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.139346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.139408 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.139504 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:29.140550 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.141581 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.143203 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.143613 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.143757 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.145666 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:29.145675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:29.145682 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:29.145725 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:29.149321 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.150120 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.150127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.150131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.150256 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.150609 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.150615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.150619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.151009 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:29.151158 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.151551 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.151758 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:40:29.152323 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.138311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.1.53.67:52873)
0927 22:40:29.139038 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.139060 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.139198 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.139531 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.139536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.140016 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:29.140417 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.140428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.141737 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.141743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.143065 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.143076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.143800 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.143806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.144048 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.144745 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:29.144762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:29.145470 (+ 708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:29.149490 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:29.149495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:29.149499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:29.150006 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:29.150361 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:29.150367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:29.150379 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:29.150501 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:29.151118 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:29.151123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:29.151308 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.151460 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.151785 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":720,"threads_started":1}
I20260927 22:40:29.168377 3326 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39031
I20260927 22:40:29.168421 3460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39031 every 8 connection(s)
I20260927 22:40:29.169652 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:29.171996 3461 heartbeater.cc:347] Connected to a master server at 127.1.53.126:40867
I20260927 22:40:29.172199 3461 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:29.172938 3461 heartbeater.cc:511] Master 127.1.53.126:40867 requested a full tablet report, sending...
I20260927 22:40:29.174440 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 3326
I20260927 22:40:29.174472 2976 ts_manager.cc:194] Registered new tserver with Master: 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.174629 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:29.175164 1237 external_mini_cluster.cc:1428] Setting key ccda012aae3390216173ee279b75351d
I20260927 22:40:29.175650 2976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:36667
I20260927 22:40:29.178303 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:40:29 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:40:29 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:29.225610 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.211055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.1.53.126:40867)
0927 22:40:29.211294 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.211302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.211480 (+ 178us) server_negotiation.cc:407] Connection header received
0927 22:40:29.211724 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.211729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.211796 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.211906 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:29.213501 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.214316 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.215672 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.216133 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.216302 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.220369 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:29.220375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:29.220379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:29.220424 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:29.223049 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.223650 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.223656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.223660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.223794 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.224156 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.224161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.224164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.224496 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:29.224732 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.225046 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.225253 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:40:29.229820 2976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:29.236780 2976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:29.272413 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.249651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.1.53.66:33125)
0927 22:40:29.262526 (+ 12875us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.262541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.262586 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:40:29.262724 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.262736 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.263107 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.264615 (+ 1508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.264859 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.266849 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.270311 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.271210 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.271370 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.271604 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.271926 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.272085 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12622,"thread_start_us":13664,"threads_started":1}
I20260927 22:40:29.273391 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.249434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33125 (local address 127.0.0.1:33622)
0927 22:40:29.250111 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.250157 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.250315 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.263699 (+ 13384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.263711 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.263759 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.264306 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.264324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.267044 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.267053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.270133 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.270148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.271430 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.271437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.271676 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.271724 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.271859 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.272049 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":28,"thread_start_us":308,"threads_started":1}
I20260927 22:40:29.275738 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.249468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.0.0.1:54660)
0927 22:40:29.266651 (+ 17183us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.266675 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.266820 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.268011 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.268015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.268039 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.268620 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.268633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.270344 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.270348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.272734 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.272748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.274825 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.274832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.275059 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.275084 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.275200 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.275340 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16813,"mutex_wait_us":1,"thread_start_us":18958,"threads_started":1}
I20260927 22:40:29.275775 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.250237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.1.53.65:38729)
0927 22:40:29.267350 (+ 17113us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.267358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.267382 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:40:29.267521 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.267531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.267789 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.268032 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.268841 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.270171 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.272936 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.274485 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.274618 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.274841 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.275269 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.275467 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16970,"thread_start_us":16836,"threads_started":1}
I20260927 22:40:29.277225 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.252309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39031 (local address 127.0.0.1:50678)
0927 22:40:29.268730 (+ 16421us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.268755 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.268875 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.269461 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.269465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.269489 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.269900 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.269910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.272174 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.272185 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.274456 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.274467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.276372 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.276378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.276555 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.276602 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.276725 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.276863 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16271,"thread_start_us":14053,"threads_started":1}
I20260927 22:40:29.277233 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.252720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.1.53.67:39031)
0927 22:40:29.265472 (+ 12752us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.265485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.268886 (+ 3401us) server_negotiation.cc:407] Connection header received
0927 22:40:29.269012 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.269020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.269249 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.269466 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.270124 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.271984 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.274619 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.276148 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.276299 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.276511 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.276783 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.276933 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12489,"thread_start_us":12764,"threads_started":1}
I20260927 22:40:29.277760 3252 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6ead14d0df4d8ca656cade978d77f6 (DEFAULT_TABLE table=test-table [id=47dc3b94ab274ee280ddbea2352477ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:29.279214 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ead14d0df4d8ca656cade978d77f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.279570 3113 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6ead14d0df4d8ca656cade978d77f6 (DEFAULT_TABLE table=test-table [id=47dc3b94ab274ee280ddbea2352477ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:29.281064 3113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ead14d0df4d8ca656cade978d77f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.281016 3391 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6ead14d0df4d8ca656cade978d77f6 (DEFAULT_TABLE table=test-table [id=47dc3b94ab274ee280ddbea2352477ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:29.282409 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ead14d0df4d8ca656cade978d77f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.290745 3480 tablet_bootstrap.cc:492] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Bootstrap starting.
I20260927 22:40:29.293730 3480 tablet_bootstrap.cc:654] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.294713 3480 log.cc:824] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:29.297427 3483 tablet_bootstrap.cc:492] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Bootstrap starting.
I20260927 22:40:29.297523 3482 tablet_bootstrap.cc:492] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Bootstrap starting.
I20260927 22:40:29.298455 3480 tablet_bootstrap.cc:492] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: No bootstrap required, opened a new log
I20260927 22:40:29.298753 3480 ts_tablet_manager.cc:1397] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:40:29.301105 3482 tablet_bootstrap.cc:654] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.301313 3483 tablet_bootstrap.cc:654] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.302204 3482 log.cc:824] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:29.302419 3483 log.cc:824] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:29.305128 3483 tablet_bootstrap.cc:492] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: No bootstrap required, opened a new log
I20260927 22:40:29.305239 3482 tablet_bootstrap.cc:492] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: No bootstrap required, opened a new log
I20260927 22:40:29.305491 3482 ts_tablet_manager.cc:1397] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:40:29.305496 3483 ts_tablet_manager.cc:1397] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:40:29.309186 3480 raft_consensus.cc:348] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.309645 3480 raft_consensus.cc:374] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.309748 3480 raft_consensus.cc:729] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.310412 3480 consensus_queue.cc:260] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.311615 3322 heartbeater.cc:503] Master 127.1.53.126:40867 was elected leader, sending a full tablet report...
I20260927 22:40:29.312315 3480 ts_tablet_manager.cc:1428] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Time spent starting tablet: real 0.013s user 0.003s sys 0.011s
I20260927 22:40:29.315227 3482 raft_consensus.cc:348] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.315608 3482 raft_consensus.cc:374] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.315735 3482 raft_consensus.cc:729] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.315564 3483 raft_consensus.cc:348] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.316054 3483 raft_consensus.cc:374] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.316184 3483 raft_consensus.cc:729] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1897cbf984014c6599441d06952a1c72, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.316547 3482 consensus_queue.cc:260] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.316871 3483 consensus_queue.cc:260] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.318598 3482 ts_tablet_manager.cc:1428] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:40:29.320175 3483 ts_tablet_manager.cc:1428] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260927 22:40:29.320423 3461 heartbeater.cc:503] Master 127.1.53.126:40867 was elected leader, sending a full tablet report...
I20260927 22:40:29.335815 3486 raft_consensus.cc:482] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:29.336169 3486 raft_consensus.cc:504] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.337857 3486 leader_election.cc:288] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.344944 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.338422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:44317 (local address 127.1.53.67:39031)
0927 22:40:29.338640 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.338649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.339420 (+ 771us) server_negotiation.cc:407] Connection header received
0927 22:40:29.339744 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.339748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.339804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.340009 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.340640 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.341308 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.343171 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.343939 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.344035 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.344160 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.344518 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.344649 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:40:29.344955 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.338344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:45885 (local address 127.1.53.65:38729)
0927 22:40:29.338588 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.338598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.339501 (+ 903us) server_negotiation.cc:407] Connection header received
0927 22:40:29.339666 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.339674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.339766 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.339835 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.340533 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.341283 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.343225 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.343981 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.344147 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.344267 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.344602 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.344729 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:40:29.345055 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.338340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.1.53.66:45885)
0927 22:40:29.339403 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.339418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.339538 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.339882 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.339887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.339910 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.340366 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.340384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.341445 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.341452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.343052 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.343064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.344272 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.344280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.344424 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.344470 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.344565 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.344675 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":45,"thread_start_us":464,"threads_started":1}
I20260927 22:40:29.345223 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.338292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39031 (local address 127.1.53.66:44317)
0927 22:40:29.339306 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.339323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.339594 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.340040 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.340043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.340065 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.340503 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.340516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.341490 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.341494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.343024 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.343038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.344173 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.344179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.344362 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.344385 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.344467 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.344588 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":44,"spinlock_wait_cycles":39680,"thread_start_us":636,"threads_started":1}
I20260927 22:40:29.346102 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ead14d0df4d8ca656cade978d77f6" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9802a3edc34b489ec509361e74724d" is_pre_election: true
I20260927 22:40:29.346221 3416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ead14d0df4d8ca656cade978d77f6" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72" is_pre_election: true
I20260927 22:40:29.346728 3133 raft_consensus.cc:2454] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 0.
I20260927 22:40:29.346727 3416 raft_consensus.cc:2454] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 0.
I20260927 22:40:29.347499 3205 leader_election.cc:302] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72, 3b80662cc6384b1692d189fce3092f9c; no voters:
I20260927 22:40:29.347905 3486 raft_consensus.cc:2781] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:29.348074 3486 raft_consensus.cc:482] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:29.348212 3486 raft_consensus.cc:3037] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.350505 3486 raft_consensus.cc:504] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.351143 3486 leader_election.cc:288] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 election: Requested vote from peers 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.351626 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ead14d0df4d8ca656cade978d77f6" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9802a3edc34b489ec509361e74724d"
I20260927 22:40:29.351683 3416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ead14d0df4d8ca656cade978d77f6" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72"
I20260927 22:40:29.351831 3133 raft_consensus.cc:3037] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.352010 3416 raft_consensus.cc:3037] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.354175 3133 raft_consensus.cc:2454] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 1.
I20260927 22:40:29.354597 3204 leader_election.cc:302] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 3b80662cc6384b1692d189fce3092f9c; no voters:
I20260927 22:40:29.354907 3486 raft_consensus.cc:2781] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:29.354939 3416 raft_consensus.cc:2454] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 1.
I20260927 22:40:29.355433 3486 raft_consensus.cc:686] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 1 LEADER]: Becoming Leader. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Running, Role: LEADER
I20260927 22:40:29.355914 3486 consensus_queue.cc:237] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.361270 2976 catalog_manager.cc:5495] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b80662cc6384b1692d189fce3092f9c (127.1.53.66). New cstate: current_term: 1 leader_uuid: "3b80662cc6384b1692d189fce3092f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } health_report { overall_health: UNKNOWN } } }
W20260927 22:40:29.380015 3184 tablet.cc:2369] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:40:29.395455 3323 tablet.cc:2369] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:40:29 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548827, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:29.414319 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.399134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.1.53.126:40867)
0927 22:40:29.399379 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.399385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.400110 (+ 725us) server_negotiation.cc:407] Connection header received
0927 22:40:29.400363 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.400368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.400457 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.400590 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:29.402302 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.403126 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.404471 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.404942 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.405043 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.409018 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:29.409023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:29.409027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:29.409055 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:29.411442 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.412341 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.412345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.412348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.412482 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:29.412925 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:29.412929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:29.412931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:29.413231 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:29.413365 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.413819 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.414032 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:40:29.414322 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.399127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40867 (local address 127.0.0.1:55402)
0927 22:40:29.399980 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.400004 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.400193 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.400738 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.400741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.401866 (+ 1125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:29.402126 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.402136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.403272 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.403276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.404227 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.404268 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.405117 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.405121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.405238 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.405935 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:29.405952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:29.408818 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:29.411583 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:29.411593 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:29.411599 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:29.412096 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:29.412590 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:29.412593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:29.412596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:29.412736 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:29.413377 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:29.413384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:29.413533 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.413675 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.413938 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":472,"threads_started":1}
I20260927 22:40:29.419937 2976 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:55402:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:40:29.421001 2976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 22:40:29.421298 3462 tablet.cc:2369] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:40:29.428220 3113 tablet_service.cc:1467] Processing CreateTablet for tablet 871a62cebbbf45bf88e0c5ff3970ee95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=541141c9a7b64f3eaaea596fa340e974]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:40:29.428223 3252 tablet_service.cc:1467] Processing CreateTablet for tablet 871a62cebbbf45bf88e0c5ff3970ee95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=541141c9a7b64f3eaaea596fa340e974]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:40:29.428359 3391 tablet_service.cc:1467] Processing CreateTablet for tablet 871a62cebbbf45bf88e0c5ff3970ee95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=541141c9a7b64f3eaaea596fa340e974]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:40:29.429170 3113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871a62cebbbf45bf88e0c5ff3970ee95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.429181 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871a62cebbbf45bf88e0c5ff3970ee95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.429181 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871a62cebbbf45bf88e0c5ff3970ee95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.435263 3482 tablet_bootstrap.cc:492] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: Bootstrap starting.
I20260927 22:40:29.436115 3480 tablet_bootstrap.cc:492] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: Bootstrap starting.
I20260927 22:40:29.436942 3482 tablet_bootstrap.cc:654] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.437744 3480 tablet_bootstrap.cc:654] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.438807 3483 tablet_bootstrap.cc:492] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: Bootstrap starting.
I20260927 22:40:29.439241 3482 tablet_bootstrap.cc:492] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: No bootstrap required, opened a new log
I20260927 22:40:29.439378 3482 ts_tablet_manager.cc:1397] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:40:29.440114 3483 tablet_bootstrap.cc:654] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.440004 3482 raft_consensus.cc:348] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.440199 3482 raft_consensus.cc:374] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.440344 3482 raft_consensus.cc:729] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.440603 3482 consensus_queue.cc:260] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.441321 3482 ts_tablet_manager.cc:1428] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:40:29.441087 3487 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.441721 3487 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.442755 3483 tablet_bootstrap.cc:492] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: No bootstrap required, opened a new log
I20260927 22:40:29.442922 3483 ts_tablet_manager.cc:1397] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260927 22:40:29.442973 3487 raft_consensus.cc:482] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:29.443236 3480 tablet_bootstrap.cc:492] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: No bootstrap required, opened a new log
I20260927 22:40:29.443125 3487 raft_consensus.cc:504] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.443378 3480 ts_tablet_manager.cc:1397] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 22:40:29.443655 3483 raft_consensus.cc:348] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.443878 3483 raft_consensus.cc:374] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.443979 3483 raft_consensus.cc:729] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1897cbf984014c6599441d06952a1c72, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.444136 3480 raft_consensus.cc:348] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.444224 3483 consensus_queue.cc:260] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.444401 3480 raft_consensus.cc:374] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.444469 3480 raft_consensus.cc:729] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.444726 3480 consensus_queue.cc:260] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.444913 3487 leader_election.cc:288] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.444741 3488 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.445079 3488 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.445322 3480 ts_tablet_manager.cc:1428] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:40:29.445360 3486 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.445590 3486 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.446429 3483 ts_tablet_manager.cc:1428] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:40:29.452474 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.445864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:57397 (local address 127.1.53.67:39031)
0927 22:40:29.446263 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.446269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.446297 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:40:29.446363 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.446368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.446476 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.446574 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.447253 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.448416 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.450683 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.451506 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.451644 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.451751 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.452043 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.452161 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260927 22:40:29.452497 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.445438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39031 (local address 127.1.53.65:57397)
0927 22:40:29.446101 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.446121 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.446241 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.446619 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.446624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.446648 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.447060 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.447077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.448854 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.448859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.450541 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.450552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.451715 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.451722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.451869 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.451897 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.452015 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.452143 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":1910,"threads_started":1}
I20260927 22:40:29.453068 3416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a62cebbbf45bf88e0c5ff3970ee95" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72" is_pre_election: true
I20260927 22:40:29.453326 3416 raft_consensus.cc:2454] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 0.
I20260927 22:40:29.453923 3065 leader_election.cc:302] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72; no voters:
I20260927 22:40:29.454437 3487 raft_consensus.cc:2781] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:29.454571 3487 raft_consensus.cc:482] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:29.454685 3487 raft_consensus.cc:3037] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.457063 3487 raft_consensus.cc:504] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.457715 3487 leader_election.cc:288] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 election: Requested vote from peers 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.458137 3416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a62cebbbf45bf88e0c5ff3970ee95" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72"
I20260927 22:40:29.458328 3416 raft_consensus.cc:3037] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.459411 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.446843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:55985 (local address 127.1.53.66:33125)
0927 22:40:29.447071 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.447079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.453219 (+ 6140us) server_negotiation.cc:407] Connection header received
0927 22:40:29.453436 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.453445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.453568 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.453680 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:29.454362 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.455635 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.457409 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.458532 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.458677 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.458784 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.459042 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.459164 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:40:29.459410 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.446009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33125 (local address 127.1.53.65:55985)
0927 22:40:29.453115 (+ 7106us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.453132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.453234 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.453875 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.453882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.453937 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:29.454177 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.454191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.455818 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.455822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.457166 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.457177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.458706 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.458712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.458887 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.458916 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.459019 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.459148 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7004,"thread_start_us":9711,"threads_started":1}
I20260927 22:40:29.460386 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a62cebbbf45bf88e0c5ff3970ee95" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b80662cc6384b1692d189fce3092f9c" is_pre_election: true
I20260927 22:40:29.460512 3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a62cebbbf45bf88e0c5ff3970ee95" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b80662cc6384b1692d189fce3092f9c"
I20260927 22:40:29.460803 3272 raft_consensus.cc:2454] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 0.
I20260927 22:40:29.461141 3416 raft_consensus.cc:2454] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 1.
I20260927 22:40:29.461827 3065 leader_election.cc:302] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72; no voters:
I20260927 22:40:29.462143 3487 raft_consensus.cc:2781] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:29.462602 3487 raft_consensus.cc:686] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [term 1 LEADER]: Becoming Leader. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Running, Role: LEADER
I20260927 22:40:29.463106 3487 consensus_queue.cc:237] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.466727 3504 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: TxnStatusTablet state changed. Reason: New leader 0f9802a3edc34b489ec509361e74724d. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.467137 3504 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:40:29.467934 3506 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:40:29.468189 3506 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:40:29.468624 2976 catalog_manager.cc:5495] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f9802a3edc34b489ec509361e74724d (127.1.53.65). New cstate: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260927 22:40:29.495260 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.489600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.0.0.1:54668)
0927 22:40:29.489842 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.489858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.490017 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.490512 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.490516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.490529 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:29.490718 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.490726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.491825 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.491833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.492907 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.492918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.493673 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.493679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.493855 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.493903 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.494723 (+ 820us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.494744 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.494824 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.494958 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:40:29.495260 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.489574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.1.53.65:38729)
0927 22:40:29.489744 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.489751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.489926 (+ 175us) server_negotiation.cc:407] Connection header received
0927 22:40:29.490220 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.490224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.490335 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.490484 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:29.490844 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.491650 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.493039 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.493483 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.493647 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.494100 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.494608 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.494757 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.494842 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.494920 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:40:29.503160 3271 raft_consensus.cc:3037] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.503585 3416 raft_consensus.cc:1261] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Refusing update from remote peer 0f9802a3edc34b489ec509361e74724d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:29.504498 3504 consensus_queue.cc:1035] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:29.506196 3271 raft_consensus.cc:1261] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Refusing update from remote peer 0f9802a3edc34b489ec509361e74724d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:29.506910 3504 consensus_queue.cc:1035] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:29.534019 3486 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: TxnStatusTablet state changed. Reason: New leader 0f9802a3edc34b489ec509361e74724d. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.534396 3486 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.534530 3488 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: TxnStatusTablet state changed. Reason: New leader 0f9802a3edc34b489ec509361e74724d. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.534803 3488 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.537213 3509 mvcc.cc:205] Tried to move back new op lower bound from 7334088005636579328 to 7334088005484613632. Current Snapshot: MvccSnapshot[applied={T|T < 7334088005636579328}]
I20260927 22:40:29.539122 3511 mvcc.cc:205] Tried to move back new op lower bound from 7334088005636579328 to 7334088005484613632. Current Snapshot: MvccSnapshot[applied={T|T < 7334088005636579328}]
I20260927 22:40:29.540472 3486 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.540814 3486 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 3b80662cc6384b1692d189fce3092f9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.549470 3504 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.549758 3504 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:40:29.553043 3504 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.553283 3504 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 0f9802a3edc34b489ec509361e74724d: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:40:29.553880 3488 tablet_replica.cc:439] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } } }
I20260927 22:40:29.554152 3488 tablet_replica.cc:442] T 871a62cebbbf45bf88e0c5ff3970ee95 P 1897cbf984014c6599441d06952a1c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:40:29.566350 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.559275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.1.53.66:33125)
0927 22:40:29.559437 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.559442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.559612 (+ 170us) server_negotiation.cc:407] Connection header received
0927 22:40:29.559792 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.559796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.559892 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.560022 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:29.560481 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.561228 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.563472 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.563959 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.564112 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.564767 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.565532 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.565656 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.565920 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.565994 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:40:29.586222 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.579446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:54251 (local address 127.1.53.65:38729)
0927 22:40:29.579677 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.579686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.580296 (+ 610us) server_negotiation.cc:407] Connection header received
0927 22:40:29.580601 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.580605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.580705 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.580784 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:29.581382 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.582153 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.584421 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.584860 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.584976 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.585586 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.585686 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.585754 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.585934 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.585997 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:40:29.586488 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.579411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.1.53.66:54251)
0927 22:40:29.580144 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.580172 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.580381 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.580831 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.580837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.580851 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:29.581194 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.581207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.582289 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.582295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.584292 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.584309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.585051 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.585059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.585257 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.585334 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.585790 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.585802 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.585911 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.586070 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":382,"threads_started":1}
I20260927 22:40:29.595957 3416 raft_consensus.cc:1261] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Refusing update from remote peer 3b80662cc6384b1692d189fce3092f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:29.596387 3133 raft_consensus.cc:1261] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Refusing update from remote peer 3b80662cc6384b1692d189fce3092f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:29.596621 3491 consensus_queue.cc:1035] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:29.597128 3486 consensus_queue.cc:1035] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:29.637305 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.629896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:42681 (local address 127.1.53.66:33125)
0927 22:40:29.630076 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:29.630083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:29.631041 (+ 958us) server_negotiation.cc:407] Connection header received
0927 22:40:29.631252 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:29.631258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:29.631344 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:29.631438 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:29.631981 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.633179 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.635016 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.635835 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.635980 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.636613 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.636734 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.636837 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:29.636944 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 22:40:29.637024 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:40:29.637501 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:29.630414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33125 (local address 127.1.53.65:42681)
0927 22:40:29.630939 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:29.630955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:29.631120 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:29.631467 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:29.631471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:29.631486 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:29.631838 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.631848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.633335 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.633340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.634876 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:29.634887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:29.636085 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:29.636090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:29.636218 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:29.636359 (+ 141us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:29.636836 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:29.636848 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:40:29.636926 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 22:40:29.637091 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260927 22:40:29.691591 2975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:29.691895 2975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:29.696303 2975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Sending DeleteTablet for 3 replicas of tablet 5e6ead14d0df4d8ca656cade978d77f6
I20260927 22:40:29.697505 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6ead14d0df4d8ca656cade978d77f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:29 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:33622
I20260927 22:40:29.698033 3391 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6ead14d0df4d8ca656cade978d77f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:29 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:50678
I20260927 22:40:29.700541 2975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:29.701262 2975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:29.702785 3113 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6ead14d0df4d8ca656cade978d77f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:29 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54660
I20260927 22:40:29.704710 3529 tablet_replica.cc:330] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: stopping tablet replica
I20260927 22:40:29.705180 3529 raft_consensus.cc:2229] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:29.705560 3529 raft_consensus.cc:2258] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:29.709690 3112 tablet_service.cc:1467] Processing CreateTablet for tablet 71e488718a924a5d9e594d6dd7d7f1bf (DEFAULT_TABLE table=test-table [id=756e7890d61246b19551d0cb86a1017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:29.710000 3391 tablet_service.cc:1467] Processing CreateTablet for tablet 71e488718a924a5d9e594d6dd7d7f1bf (DEFAULT_TABLE table=test-table [id=756e7890d61246b19551d0cb86a1017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:29.710300 3112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e488718a924a5d9e594d6dd7d7f1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.710415 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e488718a924a5d9e594d6dd7d7f1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.711409 3530 tablet_replica.cc:330] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: stopping tablet replica
I20260927 22:40:29.712718 3530 raft_consensus.cc:2229] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:29.713493 3530 raft_consensus.cc:2258] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:29.714798 3528 tablet_replica.cc:330] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: stopping tablet replica
I20260927 22:40:29.716740 3530 ts_tablet_manager.cc:1905] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:29.715438 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 71e488718a924a5d9e594d6dd7d7f1bf (DEFAULT_TABLE table=test-table [id=756e7890d61246b19551d0cb86a1017a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:29.717339 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e488718a924a5d9e594d6dd7d7f1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:29.717377 3528 raft_consensus.cc:2229] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:29.718132 3528 raft_consensus.cc:2258] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:29.720479 3483 tablet_bootstrap.cc:492] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Bootstrap starting.
I20260927 22:40:29.722137 3483 tablet_bootstrap.cc:654] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.723330 3529 ts_tablet_manager.cc:1905] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:29.724853 3482 tablet_bootstrap.cc:492] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Bootstrap starting.
I20260927 22:40:29.726495 3482 tablet_bootstrap.cc:654] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.729642 3528 ts_tablet_manager.cc:1905] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:29.730235 3480 tablet_bootstrap.cc:492] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Bootstrap starting.
I20260927 22:40:29.732074 3480 tablet_bootstrap.cc:654] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:29.735474 3529 ts_tablet_manager.cc:1918] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:29.735661 3529 log.cc:1196] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/5e6ead14d0df4d8ca656cade978d77f6
I20260927 22:40:29.736191 3529 ts_tablet_manager.cc:1939] T 5e6ead14d0df4d8ca656cade978d77f6 P 1897cbf984014c6599441d06952a1c72: Deleting consensus metadata
I20260927 22:40:29.738969 2962 catalog_manager.cc:4841] TS 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031): tablet 5e6ead14d0df4d8ca656cade978d77f6 (table test-table [id=47dc3b94ab274ee280ddbea2352477ac]) successfully deleted
I20260927 22:40:29.741729 3482 tablet_bootstrap.cc:492] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: No bootstrap required, opened a new log
I20260927 22:40:29.741918 3482 ts_tablet_manager.cc:1397] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260927 22:40:29.742707 3482 raft_consensus.cc:348] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.742988 3482 raft_consensus.cc:374] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.743080 3482 raft_consensus.cc:729] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.743314 3482 consensus_queue.cc:260] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.743937 3482 ts_tablet_manager.cc:1428] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:40:29.753199 3480 tablet_bootstrap.cc:492] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: No bootstrap required, opened a new log
I20260927 22:40:29.753463 3480 ts_tablet_manager.cc:1397] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.011s
I20260927 22:40:29.754468 3480 raft_consensus.cc:348] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.754806 3480 raft_consensus.cc:374] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.754957 3480 raft_consensus.cc:729] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.755309 3480 consensus_queue.cc:260] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.757856 3530 ts_tablet_manager.cc:1918] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:29.758096 3530 log.cc:1196] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/5e6ead14d0df4d8ca656cade978d77f6
I20260927 22:40:29.758541 3480 ts_tablet_manager.cc:1428] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260927 22:40:29.758780 3530 ts_tablet_manager.cc:1939] T 5e6ead14d0df4d8ca656cade978d77f6 P 0f9802a3edc34b489ec509361e74724d: Deleting consensus metadata
I20260927 22:40:29.760425 3528 ts_tablet_manager.cc:1918] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:29.760637 3528 log.cc:1196] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/5e6ead14d0df4d8ca656cade978d77f6
I20260927 22:40:29.761199 3528 ts_tablet_manager.cc:1939] T 5e6ead14d0df4d8ca656cade978d77f6 P 3b80662cc6384b1692d189fce3092f9c: Deleting consensus metadata
I20260927 22:40:29.762931 3483 tablet_bootstrap.cc:492] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: No bootstrap required, opened a new log
I20260927 22:40:29.762885 2960 catalog_manager.cc:4841] TS 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125): tablet 5e6ead14d0df4d8ca656cade978d77f6 (table test-table [id=47dc3b94ab274ee280ddbea2352477ac]) successfully deleted
I20260927 22:40:29.763180 3483 ts_tablet_manager.cc:1397] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.010s
I20260927 22:40:29.763208 2961 catalog_manager.cc:4841] TS 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729): tablet 5e6ead14d0df4d8ca656cade978d77f6 (table test-table [id=47dc3b94ab274ee280ddbea2352477ac]) successfully deleted
I20260927 22:40:29.764057 3483 raft_consensus.cc:348] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.764329 3483 raft_consensus.cc:374] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:29.764413 3483 raft_consensus.cc:729] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1897cbf984014c6599441d06952a1c72, State: Initialized, Role: FOLLOWER
I20260927 22:40:29.764689 3483 consensus_queue.cc:260] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.765262 3483 ts_tablet_manager.cc:1428] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:40:29.817806 3486 raft_consensus.cc:482] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:29.818043 3486 raft_consensus.cc:504] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.818845 3486 leader_election.cc:288] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.819495 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e488718a924a5d9e594d6dd7d7f1bf" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9802a3edc34b489ec509361e74724d" is_pre_election: true
I20260927 22:40:29.819790 3133 raft_consensus.cc:2454] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 0.
I20260927 22:40:29.819730 3415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e488718a924a5d9e594d6dd7d7f1bf" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72" is_pre_election: true
I20260927 22:40:29.819973 3415 raft_consensus.cc:2454] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 0.
I20260927 22:40:29.820302 3204 leader_election.cc:302] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 3b80662cc6384b1692d189fce3092f9c; no voters:
I20260927 22:40:29.820858 3486 raft_consensus.cc:2781] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:29.820981 3486 raft_consensus.cc:482] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:29.821089 3486 raft_consensus.cc:3037] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.824272 3486 raft_consensus.cc:504] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.825392 3486 leader_election.cc:288] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 election: Requested vote from peers 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:29.825525 3415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e488718a924a5d9e594d6dd7d7f1bf" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72"
I20260927 22:40:29.825757 3415 raft_consensus.cc:3037] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.825747 3133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e488718a924a5d9e594d6dd7d7f1bf" candidate_uuid: "3b80662cc6384b1692d189fce3092f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9802a3edc34b489ec509361e74724d"
I20260927 22:40:29.825970 3133 raft_consensus.cc:3037] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:29.828835 3415 raft_consensus.cc:2454] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 1.
I20260927 22:40:29.829298 3205 leader_election.cc:302] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1897cbf984014c6599441d06952a1c72, 3b80662cc6384b1692d189fce3092f9c; no voters:
I20260927 22:40:29.829308 3133 raft_consensus.cc:2454] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b80662cc6384b1692d189fce3092f9c in term 1.
I20260927 22:40:29.829758 3486 raft_consensus.cc:2781] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:29.829946 3486 raft_consensus.cc:686] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 1 LEADER]: Becoming Leader. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Running, Role: LEADER
I20260927 22:40:29.830259 3486 consensus_queue.cc:237] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:29.833221 2975 catalog_manager.cc:5495] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b80662cc6384b1692d189fce3092f9c (127.1.53.66). New cstate: current_term: 1 leader_uuid: "3b80662cc6384b1692d189fce3092f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260927 22:40:29.873916 3415 raft_consensus.cc:1261] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Refusing update from remote peer 3b80662cc6384b1692d189fce3092f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:29.874034 3133 raft_consensus.cc:1261] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Refusing update from remote peer 3b80662cc6384b1692d189fce3092f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:29.874616 3491 consensus_queue.cc:1035] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:29.875118 3491 consensus_queue.cc:1035] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:29.892035 3510 mvcc.cc:205] Tried to move back new op lower bound from 7334088007157768192 to 7334088006987223040. Current Snapshot: MvccSnapshot[applied={T|T < 7334088007157768192}]
W20260927 22:40:29.922535 3511 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:40:31.538300 2975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:31.538513 2975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:31.542704 2975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Sending DeleteTablet for 3 replicas of tablet 71e488718a924a5d9e594d6dd7d7f1bf
I20260927 22:40:31.543717 3112 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e488718a924a5d9e594d6dd7d7f1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:31 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54660
I20260927 22:40:31.543752 3251 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e488718a924a5d9e594d6dd7d7f1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:31 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:33622
I20260927 22:40:31.543880 3391 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e488718a924a5d9e594d6dd7d7f1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:31 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:50678
I20260927 22:40:31.555035 3564 tablet_replica.cc:330] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: stopping tablet replica
I20260927 22:40:31.555501 3564 raft_consensus.cc:2229] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:31.555889 3564 raft_consensus.cc:2258] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:31.556057 3563 tablet_replica.cc:330] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: stopping tablet replica
I20260927 22:40:31.556391 3563 raft_consensus.cc:2229] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:31.556896 3563 raft_consensus.cc:2258] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:31.557466 3564 ts_tablet_manager.cc:1905] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:31.558285 3563 ts_tablet_manager.cc:1905] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:31.562367 3562 tablet_replica.cc:330] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: stopping tablet replica
I20260927 22:40:31.562635 3562 raft_consensus.cc:2229] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:31.562930 3562 raft_consensus.cc:2258] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:31.564095 3562 ts_tablet_manager.cc:1905] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:40:31 dist-test-slave-pp80 krb5kdc[2931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260927 22:40:31.616065 3562 ts_tablet_manager.cc:1918] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:40:31.616061 3564 ts_tablet_manager.cc:1918] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:40:31.616231 3564 log.cc:1196] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/71e488718a924a5d9e594d6dd7d7f1bf
I20260927 22:40:31.616231 3562 log.cc:1196] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/71e488718a924a5d9e594d6dd7d7f1bf
I20260927 22:40:31.616701 3562 ts_tablet_manager.cc:1939] T 71e488718a924a5d9e594d6dd7d7f1bf P 0f9802a3edc34b489ec509361e74724d: Deleting consensus metadata
I20260927 22:40:31.616701 3564 ts_tablet_manager.cc:1939] T 71e488718a924a5d9e594d6dd7d7f1bf P 1897cbf984014c6599441d06952a1c72: Deleting consensus metadata
I20260927 22:40:31.617885 3563 ts_tablet_manager.cc:1918] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:40:31.618036 3563 log.cc:1196] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/71e488718a924a5d9e594d6dd7d7f1bf
I20260927 22:40:31.618467 3563 ts_tablet_manager.cc:1939] T 71e488718a924a5d9e594d6dd7d7f1bf P 3b80662cc6384b1692d189fce3092f9c: Deleting consensus metadata
Sep 27 22:40:31 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:31.641278 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:31.620398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.1.53.126:40867)
0927 22:40:31.621481 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:31.621516 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:31.621538 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:40:31.621591 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:31.621594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:31.621653 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:31.621776 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:31.623314 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.624331 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.625629 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.626082 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.626185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:31.636060 (+ 9875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:31.636065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:31.636069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:31.636102 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:31.638771 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:31.639385 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:31.639389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:31.639392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:31.639531 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:31.639842 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:31.639845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:31.639847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:31.640136 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:31.640378 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:31.640714 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:40:31.640930 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":724,"threads_started":1}
I20260927 22:40:31.645212 2975 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:31.645947 2975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:31.652767 3112 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1cd5772f544d24a67be1727c91a1f5 (DEFAULT_TABLE table=test-table [id=789b399e7ca84b69b5b8f6dfce785e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:31.652891 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1cd5772f544d24a67be1727c91a1f5 (DEFAULT_TABLE table=test-table [id=789b399e7ca84b69b5b8f6dfce785e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:31.653124 3391 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1cd5772f544d24a67be1727c91a1f5 (DEFAULT_TABLE table=test-table [id=789b399e7ca84b69b5b8f6dfce785e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:31.653321 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1cd5772f544d24a67be1727c91a1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:31.653321 3112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1cd5772f544d24a67be1727c91a1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:31.653513 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1cd5772f544d24a67be1727c91a1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:31.657994 2961 catalog_manager.cc:4841] TS 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729): tablet 71e488718a924a5d9e594d6dd7d7f1bf (table test-table [id=756e7890d61246b19551d0cb86a1017a]) successfully deleted
I20260927 22:40:31.659101 2962 catalog_manager.cc:4841] TS 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031): tablet 71e488718a924a5d9e594d6dd7d7f1bf (table test-table [id=756e7890d61246b19551d0cb86a1017a]) successfully deleted
I20260927 22:40:31.665786 3575 tablet_bootstrap.cc:492] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Bootstrap starting.
I20260927 22:40:31.666348 3573 tablet_bootstrap.cc:492] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Bootstrap starting.
I20260927 22:40:31.667446 3574 tablet_bootstrap.cc:492] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Bootstrap starting.
I20260927 22:40:31.668478 3573 tablet_bootstrap.cc:654] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:31.669221 3575 tablet_bootstrap.cc:654] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:31.669430 3574 tablet_bootstrap.cc:654] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:31.674849 3575 tablet_bootstrap.cc:492] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: No bootstrap required, opened a new log
I20260927 22:40:31.675029 3575 ts_tablet_manager.cc:1397] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 22:40:31.676433 3575 raft_consensus.cc:348] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.676771 3575 raft_consensus.cc:374] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:31.676862 3575 raft_consensus.cc:729] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1897cbf984014c6599441d06952a1c72, State: Initialized, Role: FOLLOWER
I20260927 22:40:31.677203 3575 consensus_queue.cc:260] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.679456 3573 tablet_bootstrap.cc:492] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: No bootstrap required, opened a new log
I20260927 22:40:31.679718 3573 ts_tablet_manager.cc:1397] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260927 22:40:31.680768 3573 raft_consensus.cc:348] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.681034 3575 ts_tablet_manager.cc:1428] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 22:40:31.681151 3573 raft_consensus.cc:374] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:31.681291 3573 raft_consensus.cc:729] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Initialized, Role: FOLLOWER
I20260927 22:40:31.681607 3573 consensus_queue.cc:260] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.683246 3573 ts_tablet_manager.cc:1428] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:40:31.685941 3574 tablet_bootstrap.cc:492] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: No bootstrap required, opened a new log
I20260927 22:40:31.686100 3574 ts_tablet_manager.cc:1397] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.013s
I20260927 22:40:31.686857 3574 raft_consensus.cc:348] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.687105 3574 raft_consensus.cc:374] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:31.687211 3574 raft_consensus.cc:729] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Initialized, Role: FOLLOWER
I20260927 22:40:31.687469 3574 consensus_queue.cc:260] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.687553 2960 catalog_manager.cc:4841] TS 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125): tablet 71e488718a924a5d9e594d6dd7d7f1bf (table test-table [id=756e7890d61246b19551d0cb86a1017a]) successfully deleted
I20260927 22:40:31.687965 3574 ts_tablet_manager.cc:1428] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:40:31.784176 3541 raft_consensus.cc:482] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:31.784443 3541 raft_consensus.cc:504] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.785277 3541 leader_election.cc:288] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:31.785805 3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1cd5772f544d24a67be1727c91a1f5" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b80662cc6384b1692d189fce3092f9c" is_pre_election: true
I20260927 22:40:31.785929 3415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1cd5772f544d24a67be1727c91a1f5" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72" is_pre_election: true
I20260927 22:40:31.786113 3271 raft_consensus.cc:2454] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 0.
I20260927 22:40:31.786164 3415 raft_consensus.cc:2454] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 0.
I20260927 22:40:31.786597 3065 leader_election.cc:302] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72; no voters:
I20260927 22:40:31.786847 3541 raft_consensus.cc:2781] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:31.786990 3541 raft_consensus.cc:482] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:31.787106 3541 raft_consensus.cc:3037] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:31.825579 3541 raft_consensus.cc:504] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.827006 3541 leader_election.cc:288] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 election: Requested vote from peers 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125), 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031)
I20260927 22:40:31.827163 3415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1cd5772f544d24a67be1727c91a1f5" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1897cbf984014c6599441d06952a1c72"
I20260927 22:40:31.827170 3271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1cd5772f544d24a67be1727c91a1f5" candidate_uuid: "0f9802a3edc34b489ec509361e74724d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b80662cc6384b1692d189fce3092f9c"
I20260927 22:40:31.827397 3415 raft_consensus.cc:3037] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:31.827419 3271 raft_consensus.cc:3037] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:31.927353 3415 raft_consensus.cc:2454] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 1.
I20260927 22:40:31.927345 3271 raft_consensus.cc:2454] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9802a3edc34b489ec509361e74724d in term 1.
I20260927 22:40:31.928073 3065 leader_election.cc:302] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72; no voters:
I20260927 22:40:31.928550 3541 raft_consensus.cc:2781] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:31.928782 3541 raft_consensus.cc:686] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 1 LEADER]: Becoming Leader. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Running, Role: LEADER
I20260927 22:40:31.929072 3541 consensus_queue.cc:237] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:31.931677 2975 catalog_manager.cc:5495] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f9802a3edc34b489ec509361e74724d (127.1.53.65). New cstate: current_term: 1 leader_uuid: "0f9802a3edc34b489ec509361e74724d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260927 22:40:31.974112 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:31.969039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.1.53.65:38729)
0927 22:40:31.969594 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:31.969602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:31.969626 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:40:31.969695 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:31.969698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:31.969759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:31.969867 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:31.970236 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.970989 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.972373 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.972779 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.972912 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:31.973339 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:31.973454 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:31.973554 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:31.973710 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:40:31.973783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":295,"threads_started":1}
I20260927 22:40:31.988888 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:31.982924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:42889 (local address 127.1.53.65:38729)
0927 22:40:31.983574 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:31.983585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:31.984235 (+ 650us) server_negotiation.cc:407] Connection header received
0927 22:40:31.984478 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:31.984483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:31.984553 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:31.984658 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:31.985105 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.985828 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.987269 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.987777 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.987873 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:31.988320 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:31.988408 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:31.988473 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:31.988615 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:40:31.988672 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:40:31.989109 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:31.982946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.1.53.65:42889)
0927 22:40:31.984121 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:31.984136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:31.984340 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:31.984678 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:31.984682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:31.984692 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:31.984944 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:31.984952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.985961 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.985967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:31.987111 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:31.987123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:31.987984 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:31.987989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:31.988106 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:31.988139 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:31.988507 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:31.988515 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:40:31.988583 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 22:40:31.988678 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":886,"threads_started":1}
I20260927 22:40:31.996731 3415 raft_consensus.cc:1261] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Refusing update from remote peer 0f9802a3edc34b489ec509361e74724d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:31.996743 3271 raft_consensus.cc:1261] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Refusing update from remote peer 0f9802a3edc34b489ec509361e74724d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:31.997255 3546 consensus_queue.cc:1035] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:31.997593 3523 consensus_queue.cc:1035] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:32.042863 2976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:32.043154 2976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:32.046579 2976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Sending DeleteTablet for 3 replicas of tablet 3a1cd5772f544d24a67be1727c91a1f5
I20260927 22:40:32.047547 3251 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1cd5772f544d24a67be1727c91a1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:32 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:33622
I20260927 22:40:32.047562 3113 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1cd5772f544d24a67be1727c91a1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:32 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54660
I20260927 22:40:32.047679 3391 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1cd5772f544d24a67be1727c91a1f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:32 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:50678
I20260927 22:40:32.047875 3562 tablet_replica.cc:330] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: stopping tablet replica
I20260927 22:40:32.047868 3563 tablet_replica.cc:330] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: stopping tablet replica
I20260927 22:40:32.047945 3564 tablet_replica.cc:330] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: stopping tablet replica
I20260927 22:40:32.048082 3563 raft_consensus.cc:2229] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:32.048102 3562 raft_consensus.cc:2229] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:32.048144 3564 raft_consensus.cc:2229] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:32.048324 3563 raft_consensus.cc:2258] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:32.048353 3564 raft_consensus.cc:2258] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:32.048971 3562 raft_consensus.cc:2258] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:32.049664 3563 ts_tablet_manager.cc:1905] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:32.049747 3564 ts_tablet_manager.cc:1905] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:32.050360 3562 ts_tablet_manager.cc:1905] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:32.054683 2976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:32.055405 2976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:32.063162 3113 tablet_service.cc:1467] Processing CreateTablet for tablet be6cae179a4d46f69c698424ea2e95d0 (DEFAULT_TABLE table=test-table [id=6f41c32734cc405b8367d3ac1dc0ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:32.063298 3391 tablet_service.cc:1467] Processing CreateTablet for tablet be6cae179a4d46f69c698424ea2e95d0 (DEFAULT_TABLE table=test-table [id=6f41c32734cc405b8367d3ac1dc0ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:32.064025 3251 tablet_service.cc:1467] Processing CreateTablet for tablet be6cae179a4d46f69c698424ea2e95d0 (DEFAULT_TABLE table=test-table [id=6f41c32734cc405b8367d3ac1dc0ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:32.130556 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6cae179a4d46f69c698424ea2e95d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:32.137596 3113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6cae179a4d46f69c698424ea2e95d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:32.139415 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be6cae179a4d46f69c698424ea2e95d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:32.141417 3575 tablet_bootstrap.cc:492] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Bootstrap starting.
I20260927 22:40:32.143028 3575 tablet_bootstrap.cc:654] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:32.145013 3573 tablet_bootstrap.cc:492] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Bootstrap starting.
I20260927 22:40:32.148474 3573 tablet_bootstrap.cc:654] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:32.149473 3574 tablet_bootstrap.cc:492] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: Bootstrap starting.
I20260927 22:40:32.150987 3574 tablet_bootstrap.cc:654] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:32.158715 3575 tablet_bootstrap.cc:492] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: No bootstrap required, opened a new log
I20260927 22:40:32.158761 3574 tablet_bootstrap.cc:492] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: No bootstrap required, opened a new log
I20260927 22:40:32.158885 3575 ts_tablet_manager.cc:1397] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20260927 22:40:32.158921 3574 ts_tablet_manager.cc:1397] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260927 22:40:32.159646 3574 raft_consensus.cc:348] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.159652 3575 raft_consensus.cc:348] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.159888 3575 raft_consensus.cc:374] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:32.159885 3574 raft_consensus.cc:374] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:32.159966 3575 raft_consensus.cc:729] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1897cbf984014c6599441d06952a1c72, State: Initialized, Role: FOLLOWER
I20260927 22:40:32.159967 3574 raft_consensus.cc:729] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b80662cc6384b1692d189fce3092f9c, State: Initialized, Role: FOLLOWER
I20260927 22:40:32.160208 3574 consensus_queue.cc:260] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.160275 3575 consensus_queue.cc:260] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.160751 3574 ts_tablet_manager.cc:1428] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:40:32.160938 3575 ts_tablet_manager.cc:1428] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:40:32.162125 3573 tablet_bootstrap.cc:492] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: No bootstrap required, opened a new log
I20260927 22:40:32.162322 3573 ts_tablet_manager.cc:1397] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260927 22:40:32.163566 3573 raft_consensus.cc:348] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.163858 3573 raft_consensus.cc:374] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:32.163949 3573 raft_consensus.cc:729] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9802a3edc34b489ec509361e74724d, State: Initialized, Role: FOLLOWER
I20260927 22:40:32.164276 3573 consensus_queue.cc:260] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.165477 3573 ts_tablet_manager.cc:1428] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 22:40:32.234311 3563 ts_tablet_manager.cc:1918] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:32.234311 3564 ts_tablet_manager.cc:1918] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:32.234510 3564 log.cc:1196] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/3a1cd5772f544d24a67be1727c91a1f5
I20260927 22:40:32.234510 3563 log.cc:1196] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/3a1cd5772f544d24a67be1727c91a1f5
I20260927 22:40:32.234928 3564 ts_tablet_manager.cc:1939] T 3a1cd5772f544d24a67be1727c91a1f5 P 1897cbf984014c6599441d06952a1c72: Deleting consensus metadata
I20260927 22:40:32.234939 3563 ts_tablet_manager.cc:1939] T 3a1cd5772f544d24a67be1727c91a1f5 P 3b80662cc6384b1692d189fce3092f9c: Deleting consensus metadata
I20260927 22:40:32.235450 3562 ts_tablet_manager.cc:1918] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:40:32.235630 3562 log.cc:1196] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/3a1cd5772f544d24a67be1727c91a1f5
I20260927 22:40:32.236097 3562 ts_tablet_manager.cc:1939] T 3a1cd5772f544d24a67be1727c91a1f5 P 0f9802a3edc34b489ec509361e74724d: Deleting consensus metadata
I20260927 22:40:32.250569 2960 catalog_manager.cc:4841] TS 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125): tablet 3a1cd5772f544d24a67be1727c91a1f5 (table test-table [id=789b399e7ca84b69b5b8f6dfce785e83]) successfully deleted
I20260927 22:40:32.251927 2962 catalog_manager.cc:4841] TS 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031): tablet 3a1cd5772f544d24a67be1727c91a1f5 (table test-table [id=789b399e7ca84b69b5b8f6dfce785e83]) successfully deleted
I20260927 22:40:32.265076 3579 raft_consensus.cc:482] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:32.265327 3579 raft_consensus.cc:504] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.265605 2961 catalog_manager.cc:4841] TS 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729): tablet 3a1cd5772f544d24a67be1727c91a1f5 (table test-table [id=789b399e7ca84b69b5b8f6dfce785e83]) successfully deleted
I20260927 22:40:32.267141 3579 leader_election.cc:288] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729), 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125)
I20260927 22:40:32.285104 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.268345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:43547 (local address 127.1.53.65:38729)
0927 22:40:32.268556 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:32.268572 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:32.278680 (+ 10108us) server_negotiation.cc:407] Connection header received
0927 22:40:32.279075 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:32.279079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:32.279146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:32.279262 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:32.279737 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.280631 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.282860 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.284051 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.284233 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.284363 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:32.284662 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:40:32.284803 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:40:32.285112 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.267392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.1.53.67:43547)
0927 22:40:32.278584 (+ 11192us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:32.278602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:32.278838 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:32.279278 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:32.279283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:32.279307 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:32.279581 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.279589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.280778 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.280784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.282696 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.282707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.284281 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.284287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.284492 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.284524 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:40:32.284635 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:40:32.284772 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11076,"thread_start_us":10550,"threads_started":1}
I20260927 22:40:32.286063 3132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6cae179a4d46f69c698424ea2e95d0" candidate_uuid: "1897cbf984014c6599441d06952a1c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9802a3edc34b489ec509361e74724d" is_pre_election: true
I20260927 22:40:32.286361 3132 raft_consensus.cc:2454] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1897cbf984014c6599441d06952a1c72 in term 0.
I20260927 22:40:32.286921 3343 leader_election.cc:302] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72; no voters:
I20260927 22:40:32.287351 3579 raft_consensus.cc:2781] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:32.287478 3579 raft_consensus.cc:482] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:32.287619 3579 raft_consensus.cc:3037] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:32.288550 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.267360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33125 (local address 127.1.53.67:34621)
0927 22:40:32.278290 (+ 10930us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:32.278323 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:32.278509 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:32.280985 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:32.280990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:32.281030 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:32.281526 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.281538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.283510 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.283515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.285762 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.285777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.287656 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.287662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.287847 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.287885 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:40:32.287975 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:40:32.288117 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10769,"mutex_wait_us":57,"thread_start_us":11254,"threads_started":1}
I20260927 22:40:32.289002 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.267748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:34621 (local address 127.1.53.66:33125)
0927 22:40:32.280649 (+ 12901us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:32.280656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:32.280684 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:40:32.280755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:32.280761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:32.280838 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:32.280962 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:32.281729 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.283322 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.285985 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.287246 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.287377 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.287515 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:32.288365 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 22:40:32.288520 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12766,"thread_start_us":13358,"threads_started":1}
I20260927 22:40:32.290607 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6cae179a4d46f69c698424ea2e95d0" candidate_uuid: "1897cbf984014c6599441d06952a1c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b80662cc6384b1692d189fce3092f9c" is_pre_election: true
I20260927 22:40:32.290823 3272 raft_consensus.cc:2454] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1897cbf984014c6599441d06952a1c72 in term 0.
I20260927 22:40:32.336408 3579 raft_consensus.cc:504] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.337155 3579 leader_election.cc:288] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [CANDIDATE]: Term 1 election: Requested vote from peers 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729), 3b80662cc6384b1692d189fce3092f9c (127.1.53.66:33125)
I20260927 22:40:32.337751 3132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6cae179a4d46f69c698424ea2e95d0" candidate_uuid: "1897cbf984014c6599441d06952a1c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9802a3edc34b489ec509361e74724d"
I20260927 22:40:32.337816 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be6cae179a4d46f69c698424ea2e95d0" candidate_uuid: "1897cbf984014c6599441d06952a1c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b80662cc6384b1692d189fce3092f9c"
I20260927 22:40:32.337980 3132 raft_consensus.cc:3037] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:32.338003 3272 raft_consensus.cc:3037] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:32.341069 3272 raft_consensus.cc:2454] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1897cbf984014c6599441d06952a1c72 in term 1.
I20260927 22:40:32.341145 3132 raft_consensus.cc:2454] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1897cbf984014c6599441d06952a1c72 in term 1.
I20260927 22:40:32.341604 3343 leader_election.cc:302] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9802a3edc34b489ec509361e74724d, 1897cbf984014c6599441d06952a1c72; no voters:
I20260927 22:40:32.341926 3579 raft_consensus.cc:2781] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:32.342450 3579 raft_consensus.cc:686] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 1 LEADER]: Becoming Leader. State: Replica: 1897cbf984014c6599441d06952a1c72, State: Running, Role: LEADER
I20260927 22:40:32.343266 3579 consensus_queue.cc:237] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } }
I20260927 22:40:32.346995 2975 catalog_manager.cc:5495] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1897cbf984014c6599441d06952a1c72 (127.1.53.67). New cstate: current_term: 1 leader_uuid: "1897cbf984014c6599441d06952a1c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1897cbf984014c6599441d06952a1c72" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39031 } health_report { overall_health: HEALTHY } } }
I20260927 22:40:32.399268 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.386976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.1.53.67:39031)
0927 22:40:32.392912 (+ 5936us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:32.392917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:32.392938 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:40:32.393036 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:32.393040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:32.393118 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:32.393275 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:32.393815 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.394619 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.396547 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.396961 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.397083 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.397740 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:32.398476 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:32.398599 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:32.398782 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 22:40:32.398856 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5828,"thread_start_us":5691,"threads_started":1}
I20260927 22:40:32.418011 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.411074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:54169 (local address 127.1.53.65:38729)
0927 22:40:32.411464 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:32.411473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:32.412887 (+ 1414us) server_negotiation.cc:407] Connection header received
0927 22:40:32.413120 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:32.413124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:32.413193 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:32.413319 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:32.413803 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.414643 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.416114 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.416649 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.416782 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.417223 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:32.417357 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:32.417458 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:32.417598 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 22:40:32.417677 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260927 22:40:32.418077 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.410756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.1.53.67:54169)
0927 22:40:32.412757 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:32.412776 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:32.412961 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:32.413356 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:32.413360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:32.413375 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:32.413658 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.413666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.414787 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.414793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.415968 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.415979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.416840 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.416847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.416984 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.417046 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:32.417458 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:32.417472 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:40:32.417579 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:40:32.417717 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1810,"thread_start_us":1551,"threads_started":1}
I20260927 22:40:32.432154 3272 raft_consensus.cc:1261] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Refusing update from remote peer 1897cbf984014c6599441d06952a1c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:32.432502 3132 raft_consensus.cc:1261] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Refusing update from remote peer 1897cbf984014c6599441d06952a1c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:32.432860 3595 consensus_queue.cc:1035] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b80662cc6384b1692d189fce3092f9c" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:32.433317 3579 consensus_queue.cc:1035] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9802a3edc34b489ec509361e74724d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:32.466346 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.459861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39031 (local address 127.1.53.65:42835)
0927 22:40:32.460153 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:32.460181 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:32.460364 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:32.460986 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:32.460994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:32.461009 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:40:32.461296 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.461311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.462844 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.462848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.463937 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:32.463951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.464961 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.464967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:32.465191 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.465265 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:40:32.465766 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:40:32.465776 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:40:32.465868 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:40:32.466019 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:40:32.466918 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:32.460328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:42835 (local address 127.1.53.67:39031)
0927 22:40:32.460531 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:32.460537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:32.460557 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:40:32.460628 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:32.460633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:32.460718 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:32.460940 (+ 222us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:32.461513 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.462692 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.464099 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:32.464775 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:32.464938 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:32.465457 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:32.465609 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:32.465692 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:32.466553 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 22:40:32.466631 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 22:40:32.469331 3542 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:40:34.077715 2975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:34.078076 2975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:34.081853 2975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82dedba1cb2a4d569bce05d0850b1e86: Sending DeleteTablet for 3 replicas of tablet be6cae179a4d46f69c698424ea2e95d0
I20260927 22:40:34.082667 3113 tablet_service.cc:1514] Processing DeleteTablet for tablet be6cae179a4d46f69c698424ea2e95d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:34 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54660
I20260927 22:40:34.082669 3251 tablet_service.cc:1514] Processing DeleteTablet for tablet be6cae179a4d46f69c698424ea2e95d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:34 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:33622
I20260927 22:40:34.083370 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3047
I20260927 22:40:34.084137 3391 tablet_service.cc:1514] Processing DeleteTablet for tablet be6cae179a4d46f69c698424ea2e95d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:34 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:50678
I20260927 22:40:34.093420 3629 tablet_replica.cc:330] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: stopping tablet replica
I20260927 22:40:34.094311 3629 raft_consensus.cc:2229] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:34.094961 3629 raft_consensus.cc:2258] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:34.095507 3628 tablet_replica.cc:330] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: stopping tablet replica
I20260927 22:40:34.095769 3628 raft_consensus.cc:2229] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:34.096035 3628 raft_consensus.cc:2258] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:34.096062 3629 ts_tablet_manager.cc:1905] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:34.096662 3627 tablet_replica.cc:330] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: stopping tablet replica
I20260927 22:40:34.096923 3627 raft_consensus.cc:2229] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:34.097195 3627 raft_consensus.cc:2258] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:34.098750 3627 ts_tablet_manager.cc:1905] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:34.099656 3628 ts_tablet_manager.cc:1905] T be6cae179a4d46f69c698424ea2e95d0 P 3b80662cc6384b1692d189fce3092f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:40:34 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:34.112262 3629 ts_tablet_manager.cc:1918] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:40:34.112533 3629 log.cc:1196] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/be6cae179a4d46f69c698424ea2e95d0
I20260927 22:40:34.112555 3627 ts_tablet_manager.cc:1918] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:40:34.112668 3627 log.cc:1196] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/be6cae179a4d46f69c698424ea2e95d0
I20260927 22:40:34.113166 3629 ts_tablet_manager.cc:1939] T be6cae179a4d46f69c698424ea2e95d0 P 1897cbf984014c6599441d06952a1c72: Deleting consensus metadata
I20260927 22:40:34.113163 3627 ts_tablet_manager.cc:1939] T be6cae179a4d46f69c698424ea2e95d0 P 0f9802a3edc34b489ec509361e74724d: Deleting consensus metadata
I20260927 22:40:34.115803 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.084583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.1.53.65:38729)
0927 22:40:34.094034 (+ 9451us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:34.094041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:34.097918 (+ 3877us) server_negotiation.cc:407] Connection header received
0927 22:40:34.098083 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:34.098089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:34.098174 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:34.098295 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:34.099925 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:34.101354 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:34.102778 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:34.103212 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:34.103315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:34.107299 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:34.107313 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:34.107329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:34.107368 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:34.111755 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:34.112822 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:34.112834 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:34.112841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:34.113020 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:34.113513 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:34.113518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:34.113521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:34.113965 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:34.114279 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:34.115191 (+ 912us) server_negotiation.cc:299] Negotiation successful
0927 22:40:34.115428 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9306,"thread_start_us":9163,"threads_started":1}
W20260927 22:40:34.116575 3178 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43070
W20260927 22:40:34.117213 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3047: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:34.117410 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 3047
I20260927 22:40:34.152992 2962 catalog_manager.cc:4841] TS 1897cbf984014c6599441d06952a1c72 (127.1.53.67:39031): tablet be6cae179a4d46f69c698424ea2e95d0 (table test-table [id=6f41c32734cc405b8367d3ac1dc0ea22]) successfully deleted
I20260927 22:40:34.188624 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3187
W20260927 22:40:34.188616 2961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:38729 (error 108)
I20260927 22:40:34.190651 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.189914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.0.0.1:43072)
0927 22:40:34.190336 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:34.190444 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":207,"threads_started":1}
W20260927 22:40:34.190879 2961 catalog_manager.cc:4584] TS 0f9802a3edc34b489ec509361e74724d (127.1.53.65:38729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be6cae179a4d46f69c698424ea2e95d0: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38729: connect: Connection refused (error 111)
Sep 27 22:40:34 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:34.205699 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.189478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.1.53.66:33125)
0927 22:40:34.189941 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:34.189946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:34.190337 (+ 391us) server_negotiation.cc:407] Connection header received
0927 22:40:34.190553 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:34.190559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:34.190629 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:34.190743 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:34.192402 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:34.193141 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:34.194707 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:34.195156 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:34.195299 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:34.199748 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:34.199772 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:34.199788 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:34.199830 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:34.202744 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:34.203534 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:34.203547 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:34.203550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:34.203688 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:34.204074 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:34.204084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:34.204086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:34.204484 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:34.204798 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:34.205117 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:40:34.205340 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":229,"threads_started":1}
W20260927 22:40:34.206442 3317 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34968
W20260927 22:40:34.207142 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:34.207336 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 3187
I20260927 22:40:34.230757 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3326
I20260927 22:40:34.231026 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.230542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33125 (local address 127.0.0.1:34976)
0927 22:40:34.230739 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:34.230843 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.66:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
Sep 27 22:40:34 dist-test-slave-pp80 krb5kdc[2931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:34.244866 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.244066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.0.0.1:43076)
0927 22:40:34.244565 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:34.244672 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:40:34.245873 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.245541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.0.0.1:43092)
0927 22:40:34.245687 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:34.245750 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:40:34.246430 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.231493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.1.53.67:39031)
0927 22:40:34.232019 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:34.232023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:34.232039 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:40:34.232093 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:34.232096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:34.232151 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:34.232296 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:34.233629 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:34.234398 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:34.235671 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:34.236057 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:34.236154 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:34.240563 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:34.240585 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:34.240602 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:34.240641 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:34.243489 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:34.244129 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:34.244139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:34.244142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:34.244348 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:34.244746 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:34.244750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:34.244752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:34.245203 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:34.245544 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:34.245854 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 22:40:34.246095 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
W20260927 22:40:34.247089 3456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34622
W20260927 22:40:34.247777 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3326: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:34.247965 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 3326
I20260927 22:40:34.273970 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 2944
I20260927 22:40:34.275583 3016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:34.392329 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.275426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33125 (local address 127.0.0.1:34982)
0927 22:40:34.275642 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:34.275733 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.66:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:40:34.393051 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:34.392031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38729 (local address 127.0.0.1:43108)
0927 22:40:34.392686 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:34.392782 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"spinlock_wait_cycles":5888,"thread_start_us":450,"threads_started":1}
I20260927 22:40:34.393193 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 2944
2026-09-27T22:40:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8607 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:36 dist-test-slave-pp80 krb5kdc[3640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:36 dist-test-slave-pp80 krb5kdc[3640](info): set up 2 sockets
Sep 27 22:40:36 dist-test-slave-pp80 krb5kdc[3640](info): commencing operation
krb5kdc: starting...
W20260927 22:40:37.288686 1237 mini_kdc.cc:121] Time spent starting KDC: real 2.848s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:37 dist-test-slave-pp80 krb5kdc[3640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:37.484392 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33253
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:37.824858 3653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:37.825181 3653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:37.825359 3653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:37.833017 3653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:37.833140 3653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:37.833245 3653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:37.833338 3653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:37.833403 3653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:37.843390 3653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:37.844871 3653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:37.846239 3653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:37.858989 3658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:37.859048 3661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:37.859074 3659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:37.859534 3653 server_base.cc:1034] running on GCE node
I20260927 22:40:37.860519 3653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:37.862238 3653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:37.863459 3653 hybrid_clock.cc:648] HybridClock initialized: now 1790548837863402 us; error 53 us; skew 500 ppm
I20260927 22:40:37.863824 3653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:37 dist-test-slave-pp80 krb5kdc[3640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:37.874727 3653 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:37.876432 3653 webserver.cc:460] Webserver started at http://127.1.53.126:40565/ using document root <none> and password file <none>
I20260927 22:40:37.877180 3653 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:37.877327 3653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:37.877661 3653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:37.880788 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "3b042a811bc041ef9066a765462c5074"
format_stamp: "Formatted at 2026-09-27 22:40:37 on dist-test-slave-pp80"
server_key: "c6563d9ac981a767ee93bcde758727f4"
server_key_iv: "123855d4629e7a163cf992bc0a80b8cf"
server_key_version: "encryptionkey@0"
I20260927 22:40:37.881590 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "3b042a811bc041ef9066a765462c5074"
format_stamp: "Formatted at 2026-09-27 22:40:37 on dist-test-slave-pp80"
server_key: "c6563d9ac981a767ee93bcde758727f4"
server_key_iv: "123855d4629e7a163cf992bc0a80b8cf"
server_key_version: "encryptionkey@0"
I20260927 22:40:37.887007 3653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:37.890861 3668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:37.892472 3653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:40:37.892668 3653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "3b042a811bc041ef9066a765462c5074"
format_stamp: "Formatted at 2026-09-27 22:40:37 on dist-test-slave-pp80"
server_key: "c6563d9ac981a767ee93bcde758727f4"
server_key_iv: "123855d4629e7a163cf992bc0a80b8cf"
server_key_version: "encryptionkey@0"
I20260927 22:40:37.892881 3653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:37.916409 3653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:37.921523 3653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:37.921852 3653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:37.940964 3653 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:33253
I20260927 22:40:37.940987 3719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:33253 every 8 connection(s)
I20260927 22:40:37.942348 3653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:37.944793 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 3653
I20260927 22:40:37.945020 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:37.945676 1237 external_mini_cluster.cc:1428] Setting key ec7c17b0e3ab8d4dc4b996f45fad0dde
I20260927 22:40:37.946273 3720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:37.960206 3720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074: Bootstrap starting.
Sep 27 22:40:37 dist-test-slave-pp80 krb5kdc[3640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:37.964087 3720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:37.965564 3720 log.cc:824] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:37.969197 3720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074: No bootstrap required, opened a new log
I20260927 22:40:37.970919 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:37.947660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.1.53.126:33253)
0927 22:40:37.948560 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:37.948575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:37.948624 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:40:37.951064 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:37.951093 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:37.951618 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:37.952082 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:37.953453 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:37.955396 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:37.956735 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:37.957836 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:37.958039 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:37.961956 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:37.961972 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:37.962003 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:37.962048 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:37.965969 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:37.967074 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:37.967083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:37.967093 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:37.967287 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:37.967692 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:37.967696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:37.967699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:37.968414 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:37.968686 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:37.969105 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:40:37.969542 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":347,"threads_started":1}
I20260927 22:40:37.977814 3720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b042a811bc041ef9066a765462c5074" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33253 } }
I20260927 22:40:37.978246 3720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:37.978395 3720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b042a811bc041ef9066a765462c5074, State: Initialized, Role: FOLLOWER
I20260927 22:40:37.979002 3720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b042a811bc041ef9066a765462c5074" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33253 } }
I20260927 22:40:37.979236 3720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:37.979364 3720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:37.979563 3720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:37.981750 3720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b042a811bc041ef9066a765462c5074" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33253 } }
I20260927 22:40:37.982254 3720 leader_election.cc:302] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b042a811bc041ef9066a765462c5074; no voters:
I20260927 22:40:37.982852 3720 leader_election.cc:288] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:37.983007 3725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:37.983690 3725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [term 1 LEADER]: Becoming Leader. State: Replica: 3b042a811bc041ef9066a765462c5074, State: Running, Role: LEADER
I20260927 22:40:37.984313 3725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b042a811bc041ef9066a765462c5074" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33253 } }
I20260927 22:40:37.985124 3720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:37.987432 3726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b042a811bc041ef9066a765462c5074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b042a811bc041ef9066a765462c5074" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33253 } } }
I20260927 22:40:37.987407 3727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b042a811bc041ef9066a765462c5074. Latest consensus state: current_term: 1 leader_uuid: "3b042a811bc041ef9066a765462c5074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b042a811bc041ef9066a765462c5074" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33253 } } }
I20260927 22:40:37.987851 3726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:37.987851 3727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:37.989557 3732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:37.994692 3732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:38.004599 3732 catalog_manager.cc:1347] Generated new cluster ID: f70ce12e07a14994bf0e5b7de9c79da0
I20260927 22:40:38.004729 3732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:38.026013 3732 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:38.027344 3732 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:38.037235 3732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b042a811bc041ef9066a765462c5074: Generated new TSK 0
I20260927 22:40:38.038223 3732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:38.096412 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33253
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:38.447212 3746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:38.447595 3746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:38.447746 3746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:38.456873 3746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:38.456983 3746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:38.457098 3746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:38.466753 3746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:38.468063 3746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:38.469352 3746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:38.481956 3754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:38.482048 3752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:38.482223 3751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:38.483039 3746 server_base.cc:1034] running on GCE node
I20260927 22:40:38.483615 3746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:38.484577 3746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:38.485824 3746 hybrid_clock.cc:648] HybridClock initialized: now 1790548838485744 us; error 75 us; skew 500 ppm
I20260927 22:40:38.486126 3746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:38 dist-test-slave-pp80 krb5kdc[3640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:38.489329 3746 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:38.490782 3746 webserver.cc:460] Webserver started at http://127.1.53.65:34027/ using document root <none> and password file <none>
I20260927 22:40:38.491443 3746 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:38.491592 3746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:38.491904 3746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:38.494884 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "0906b72ffc4a4c31b313b6ef865391fe"
format_stamp: "Formatted at 2026-09-27 22:40:38 on dist-test-slave-pp80"
server_key: "c05515891096c724e6a9f47bbe848fd1"
server_key_iv: "e81344dd4fd00786b21c0b67cf1d310c"
server_key_version: "encryptionkey@0"
I20260927 22:40:38.495690 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "0906b72ffc4a4c31b313b6ef865391fe"
format_stamp: "Formatted at 2026-09-27 22:40:38 on dist-test-slave-pp80"
server_key: "c05515891096c724e6a9f47bbe848fd1"
server_key_iv: "e81344dd4fd00786b21c0b67cf1d310c"
server_key_version: "encryptionkey@0"
I20260927 22:40:38.501132 3746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:40:38.504678 3761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:38.506106 3746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:40:38.506286 3746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "0906b72ffc4a4c31b313b6ef865391fe"
format_stamp: "Formatted at 2026-09-27 22:40:38 on dist-test-slave-pp80"
server_key: "c05515891096c724e6a9f47bbe848fd1"
server_key_iv: "e81344dd4fd00786b21c0b67cf1d310c"
server_key_version: "encryptionkey@0"
I20260927 22:40:38.506497 3746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:38.541160 3746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:38.545523 3746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:38.545828 3746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:38.546792 3746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:38.548099 3746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:38.548202 3746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:38.548380 3746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:38.548458 3746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:38.582800 3746 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:41777
I20260927 22:40:38.582844 3873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:41777 every 8 connection(s)
I20260927 22:40:38.584106 3746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:38.587607 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 3746
I20260927 22:40:38.587837 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:38.588697 1237 external_mini_cluster.cc:1428] Setting key ea7f3fa33abced0ecc83de5194aea5fb
Sep 27 22:40:38 dist-test-slave-pp80 krb5kdc[3640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:38.604472 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:38.586942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:44847 (local address 127.1.53.126:33253)
0927 22:40:38.587401 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:38.587406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:38.588640 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 22:40:38.590657 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:38.590662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:38.590741 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:38.590863 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:38.592489 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:38.593260 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:38.594887 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:38.595286 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:38.595417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:38.599012 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:38.599017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:38.599020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:38.599049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:38.601506 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:38.602215 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:38.602241 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:38.602243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:38.602356 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:38.602786 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:38.602796 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:38.602798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:38.603049 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:38.603199 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:38.603881 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 22:40:38.604093 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:40:38.605515 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:38.587668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33253 (local address 127.1.53.65:44847)
0927 22:40:38.588483 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:38.588529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:38.590408 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:38.591102 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:38.591111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:38.591622 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:38.592335 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:38.592355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:38.593428 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:38.593434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:38.594742 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:38.594754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:38.595512 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:38.595521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:38.595768 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:38.596551 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:38.596584 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:38.598798 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:38.601650 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:38.601659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:38.601676 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:38.602086 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:38.602477 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:38.602482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:38.602485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:38.602665 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:38.603174 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:38.603181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:38.603517 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 22:40:38.603781 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 22:40:38.604347 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":327,"threads_started":1}
I20260927 22:40:38.607203 3874 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33253
I20260927 22:40:38.607471 3874 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:40:38.608361 3874 heartbeater.cc:511] Master 127.1.53.126:33253 requested a full tablet report, sending...
I20260927 22:40:38.610666 3685 ts_manager.cc:194] Registered new tserver with Master: 0906b72ffc4a4c31b313b6ef865391fe (127.1.53.65:41777)
I20260927 22:40:38.612149 3685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:44847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:38.633797 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33253
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:38.969489 3880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:38.969764 3880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:38.969929 3880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:38.977259 3880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:38.977370 3880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:38.977577 3880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:38.987540 3880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:38.988893 3880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:38.990171 3880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:39.003361 3886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:39.003502 3885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:39.003472 3888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:39.004122 3880 server_base.cc:1034] running on GCE node
I20260927 22:40:39.004727 3880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:39.005702 3880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:39.006940 3880 hybrid_clock.cc:648] HybridClock initialized: now 1790548839006875 us; error 57 us; skew 500 ppm
I20260927 22:40:39.007251 3880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[3640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:39.010354 3880 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:39.011834 3880 webserver.cc:460] Webserver started at http://127.1.53.66:38135/ using document root <none> and password file <none>
I20260927 22:40:39.012574 3880 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:39.012717 3880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:39.013022 3880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:39.015983 3880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "df6288522649430e93602ab0cd7bc262"
format_stamp: "Formatted at 2026-09-27 22:40:39 on dist-test-slave-pp80"
server_key: "bdd9aca2fd2d347850cfed6c044abcf6"
server_key_iv: "f1847674510e2133c350685e55bce4fe"
server_key_version: "encryptionkey@0"
I20260927 22:40:39.016821 3880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "df6288522649430e93602ab0cd7bc262"
format_stamp: "Formatted at 2026-09-27 22:40:39 on dist-test-slave-pp80"
server_key: "bdd9aca2fd2d347850cfed6c044abcf6"
server_key_iv: "f1847674510e2133c350685e55bce4fe"
server_key_version: "encryptionkey@0"
I20260927 22:40:39.022095 3880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:39.026106 3895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:39.027577 3880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:40:39.027778 3880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "df6288522649430e93602ab0cd7bc262"
format_stamp: "Formatted at 2026-09-27 22:40:39 on dist-test-slave-pp80"
server_key: "bdd9aca2fd2d347850cfed6c044abcf6"
server_key_iv: "f1847674510e2133c350685e55bce4fe"
server_key_version: "encryptionkey@0"
I20260927 22:40:39.028000 3880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:39.067955 3880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:39.072971 3880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:39.073311 3880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:39.074453 3880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:39.075969 3880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:39.076087 3880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:39.076308 3880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:39.076398 3880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:40:39.113844 3880 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:42285
I20260927 22:40:39.113893 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:42285 every 8 connection(s)
I20260927 22:40:39.115334 3880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:39.125607 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 3880
I20260927 22:40:39.125854 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:39.126313 1237 external_mini_cluster.cc:1428] Setting key 97f38688d7071e527ae5c7462e6096dc
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[3640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:39.138561 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:39.118364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:40463 (local address 127.1.53.126:33253)
0927 22:40:39.118889 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:39.118897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:39.119996 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 22:40:39.121968 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:39.121974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:39.122047 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:39.122154 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:39.123810 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.124703 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.126708 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.127269 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.127378 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:39.131543 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:39.131553 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:39.131561 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:39.131611 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:39.134854 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:39.135893 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:39.135897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:39.135899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:39.136047 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:39.136595 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:39.136599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:39.136602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:39.136887 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:39.137060 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:39.138083 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 22:40:39.138258 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
I20260927 22:40:39.140064 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:39.119004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33253 (local address 127.1.53.66:40463)
0927 22:40:39.119794 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:39.119842 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:39.121680 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:39.122407 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:39.122417 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:39.122992 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:39.123598 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:39.123616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.124892 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.124896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:39.126545 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:39.126558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.127530 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.127536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:39.127752 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:39.128644 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:39.128674 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:39.131265 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:39.135030 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:39.135044 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:39.135067 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:39.135747 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:39.136169 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:39.136177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:39.136183 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:39.136458 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:39.137077 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:39.137088 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:39.137602 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 22:40:39.137936 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 22:40:39.138544 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":312,"threads_started":1}
I20260927 22:40:39.141686 4008 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33253
I20260927 22:40:39.142040 4008 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:39.142808 4008 heartbeater.cc:511] Master 127.1.53.126:33253 requested a full tablet report, sending...
I20260927 22:40:39.144421 3685 ts_manager.cc:194] Registered new tserver with Master: df6288522649430e93602ab0cd7bc262 (127.1.53.66:42285)
I20260927 22:40:39.145242 3685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:40463
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:39.172086 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33253
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:39.503229 4014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:39.503575 4014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:39.503755 4014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:39.511363 4014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:39.511518 4014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:39.511708 4014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:39.521656 4014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:39.523012 4014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:39.524407 4014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:39.537882 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:39.537909 4022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:39.537946 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:39.538800 4014 server_base.cc:1034] running on GCE node
I20260927 22:40:39.539446 4014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:39.540540 4014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:39.541780 4014 hybrid_clock.cc:648] HybridClock initialized: now 1790548839541727 us; error 51 us; skew 500 ppm
I20260927 22:40:39.542135 4014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[3640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:39.545403 4014 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:39.546939 4014 webserver.cc:460] Webserver started at http://127.1.53.67:34611/ using document root <none> and password file <none>
I20260927 22:40:39.547631 4014 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:39.547780 4014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:39.548103 4014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:39.551141 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "c24eb8bb085149dfba91d96427c9bc46"
format_stamp: "Formatted at 2026-09-27 22:40:39 on dist-test-slave-pp80"
server_key: "dd387f0076f8303897920a0ad139b5a6"
server_key_iv: "f2dfa809c3e4b0cbcd09047838827358"
server_key_version: "encryptionkey@0"
I20260927 22:40:39.551914 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "c24eb8bb085149dfba91d96427c9bc46"
format_stamp: "Formatted at 2026-09-27 22:40:39 on dist-test-slave-pp80"
server_key: "dd387f0076f8303897920a0ad139b5a6"
server_key_iv: "f2dfa809c3e4b0cbcd09047838827358"
server_key_version: "encryptionkey@0"
I20260927 22:40:39.557025 4014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:40:39.560737 4029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:39.562333 4014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:40:39.562543 4014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "c24eb8bb085149dfba91d96427c9bc46"
format_stamp: "Formatted at 2026-09-27 22:40:39 on dist-test-slave-pp80"
server_key: "dd387f0076f8303897920a0ad139b5a6"
server_key_iv: "f2dfa809c3e4b0cbcd09047838827358"
server_key_version: "encryptionkey@0"
I20260927 22:40:39.562759 4014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:39.582156 4014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:39.586838 4014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:39.587273 4014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:39.588330 4014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:39.589669 4014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:39.589798 4014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:39.589956 4014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:39.590030 4014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:39.615756 3874 heartbeater.cc:503] Master 127.1.53.126:33253 was elected leader, sending a full tablet report...
I20260927 22:40:39.624573 4014 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:42277
I20260927 22:40:39.624629 4141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:42277 every 8 connection(s)
I20260927 22:40:39.625810 4014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:39.631687 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4014
I20260927 22:40:39.631888 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:39.632388 1237 external_mini_cluster.cc:1428] Setting key f712552a5cd21a12bdb82020fb139f8c
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[3640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:39.646576 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:39.628257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:38629 (local address 127.1.53.126:33253)
0927 22:40:39.628448 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:39.628453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:39.629697 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 22:40:39.631469 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:39.631486 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:39.631582 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:39.631705 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:39.633821 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.634815 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.636699 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.637102 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.637208 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:39.640900 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:39.640908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:39.640915 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:39.640954 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:39.643593 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:39.644307 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:39.644314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:39.644318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:39.644432 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:39.644876 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:39.644881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:39.644883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:39.645158 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:39.645323 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:39.646027 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 22:40:39.646237 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:40:39.647807 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:39.628761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33253 (local address 127.1.53.67:38629)
0927 22:40:39.629535 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:39.629571 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:39.631200 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:39.632084 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:39.632098 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:39.632771 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:39.633464 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:39.633508 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.635003 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.635009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:39.636570 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:39.636581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:39.637333 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:39.637339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:39.637550 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:39.638192 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:39.638218 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:39.640646 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:39.643735 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:39.643745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:39.643763 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:39.644159 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:39.644560 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:39.644565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:39.644569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:39.644749 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:39.645296 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:39.645304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:39.645634 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 22:40:39.645935 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 22:40:39.646457 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":326,"threads_started":1}
I20260927 22:40:39.649137 4142 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33253
I20260927 22:40:39.649439 4142 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:39.650130 4142 heartbeater.cc:511] Master 127.1.53.126:33253 requested a full tablet report, sending...
I20260927 22:40:39.651513 3685 ts_manager.cc:194] Registered new tserver with Master: c24eb8bb085149dfba91d96427c9bc46 (127.1.53.67:42277)
I20260927 22:40:39.652400 3685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:38629
I20260927 22:40:39.660351 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:40:39.690858 4010 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:40:39.688206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.1.53.126:33253)
0927 22:40:39.688443 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:39.688452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:39.688841 (+ 389us) server_negotiation.cc:407] Connection header received
0927 22:40:39.689128 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:39.689133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:39.689208 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:39.689320 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:39.690557 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49862: BlockingRecv error: recv got EOF from 127.0.0.1:49862 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:40:39.694128 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3746
W20260927 22:40:39.697197 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3746: Not authorized: Client connection negotiation failed: client connection to 127.1.53.65:41777: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:40:39.697444 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 3746
I20260927 22:40:39.713068 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3880
W20260927 22:40:39.715633 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3880: Not authorized: Client connection negotiation failed: client connection to 127.1.53.66:42285: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:40:39.715831 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 3880
I20260927 22:40:39.731145 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4014
W20260927 22:40:39.734454 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4014: Not authorized: Client connection negotiation failed: client connection to 127.1.53.67:42277: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:40:39.734615 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4014
I20260927 22:40:39.750669 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 3653
I20260927 22:40:39.752009 3715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:39.854754 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 3653
2026-09-27T22:40:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5456 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[4159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[4159](info): set up 2 sockets
Sep 27 22:40:39 dist-test-slave-pp80 krb5kdc[4159](info): commencing operation
krb5kdc: starting...
W20260927 22:40:40.980698 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:41 dist-test-slave-pp80 krb5kdc[4159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:41.181092 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33491
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:33491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:41.504662 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:41.504976 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:41.505134 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:41.512445 4172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:41.512554 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:41.512667 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:41.512771 4172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:41.512859 4172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:41.522809 4172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:33491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:41.524459 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:41.525769 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:41.538028 4178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:41.538094 4180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:41.538129 4177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:41.541931 4172 server_base.cc:1034] running on GCE node
I20260927 22:40:41.542750 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:41.544456 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:41.545701 4172 hybrid_clock.cc:648] HybridClock initialized: now 1790548841545656 us; error 38 us; skew 500 ppm
I20260927 22:40:41.546152 4172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:41 dist-test-slave-pp80 krb5kdc[4159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:41.549423 4172 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:41.551111 4172 webserver.cc:460] Webserver started at http://127.1.53.126:42957/ using document root <none> and password file <none>
I20260927 22:40:41.551880 4172 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:41.552029 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:41.552408 4172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:41.555351 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "4c7fde76fd0e481eb9260e27f8a89be4"
format_stamp: "Formatted at 2026-09-27 22:40:41 on dist-test-slave-pp80"
server_key: "531ba4bccd2d94a9251dc0e26b60f8fa"
server_key_iv: "6c8584cfe92fc82110f1a85ed07dbfbb"
server_key_version: "encryptionkey@0"
I20260927 22:40:41.556159 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "4c7fde76fd0e481eb9260e27f8a89be4"
format_stamp: "Formatted at 2026-09-27 22:40:41 on dist-test-slave-pp80"
server_key: "531ba4bccd2d94a9251dc0e26b60f8fa"
server_key_iv: "6c8584cfe92fc82110f1a85ed07dbfbb"
server_key_version: "encryptionkey@0"
I20260927 22:40:41.561563 4172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:40:41.565404 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:41.566931 4172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:40:41.567142 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "4c7fde76fd0e481eb9260e27f8a89be4"
format_stamp: "Formatted at 2026-09-27 22:40:41 on dist-test-slave-pp80"
server_key: "531ba4bccd2d94a9251dc0e26b60f8fa"
server_key_iv: "6c8584cfe92fc82110f1a85ed07dbfbb"
server_key_version: "encryptionkey@0"
I20260927 22:40:41.567354 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:41.581677 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:41.585805 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:41.586124 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:41.604981 4172 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:33491
I20260927 22:40:41.605026 4238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:33491 every 8 connection(s)
I20260927 22:40:41.606380 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:41.610054 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:41.610278 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4172
I20260927 22:40:41.610502 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:41.611136 1237 external_mini_cluster.cc:1428] Setting key 79318e96e707be830f37eac8414ad2d0
I20260927 22:40:41.624418 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4: Bootstrap starting.
Sep 27 22:40:41 dist-test-slave-pp80 krb5kdc[4159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:41.627553 4239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:41.628966 4239 log.cc:824] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:41.632200 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4: No bootstrap required, opened a new log
I20260927 22:40:41.635990 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:41.613095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.1.53.126:33491)
0927 22:40:41.613902 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:41.613917 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:41.613981 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:40:41.616606 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:41.616637 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:41.617191 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:41.617674 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:41.618993 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:41.621018 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:41.622224 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:41.623244 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:41.623413 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:41.627429 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:41.627446 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:41.627467 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:41.627519 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:41.631290 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:41.631923 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:41.631931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:41.631939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:41.632140 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:41.632644 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:41.632651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:41.632656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:41.633391 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:41.633710 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:41.634146 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 22:40:41.634601 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":351,"threads_started":1}
I20260927 22:40:41.640295 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:40:41.641921 4239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33491 } }
I20260927 22:40:41.642279 4239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:41.642400 4239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7fde76fd0e481eb9260e27f8a89be4, State: Initialized, Role: FOLLOWER
I20260927 22:40:41.642972 4239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33491 } }
I20260927 22:40:41.643198 4239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:41.643318 4239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:41.643489 4239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:41.645663 4239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33491 } }
I20260927 22:40:41.646176 4239 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c7fde76fd0e481eb9260e27f8a89be4; no voters:
I20260927 22:40:41.646816 4239 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:41.646919 4245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:41.648223 4245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [term 1 LEADER]: Becoming Leader. State: Replica: 4c7fde76fd0e481eb9260e27f8a89be4, State: Running, Role: LEADER
I20260927 22:40:41.648828 4245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33491 } }
I20260927 22:40:41.649189 4239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:41.651886 4247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7fde76fd0e481eb9260e27f8a89be4. Latest consensus state: current_term: 1 leader_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33491 } } }
I20260927 22:40:41.652311 4247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:41.652923 4246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7fde76fd0e481eb9260e27f8a89be4" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33491 } } }
I20260927 22:40:41.653237 4246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:41.655493 4254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:41.661172 4254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:41.670166 4254 catalog_manager.cc:1347] Generated new cluster ID: cc81bd3d774742bc9eebf91103c4041e
I20260927 22:40:41.670272 4254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:41.681686 4254 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:41.683555 4254 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:41.690208 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:41.682541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.1.53.126:33491)
0927 22:40:41.682760 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:41.682769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:41.683358 (+ 589us) server_negotiation.cc:407] Connection header received
0927 22:40:41.683579 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:41.683584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:41.683662 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:41.683773 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:41.684873 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:41.685609 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:41.686903 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:41.687334 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:41.687485 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:41.688656 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:41.688660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:41.688667 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:41.689364 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:41.689426 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:41.689452 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:41.689585 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:41.689722 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:40:41.689820 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:40:41.693679 4254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c7fde76fd0e481eb9260e27f8a89be4: Generated new TSK 0
I20260927 22:40:41.694751 4254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:40:41.703400 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4172
I20260927 22:40:41.704818 4234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:41.792368 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4172
2026-09-27T22:40:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1932 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:41 dist-test-slave-pp80 krb5kdc[4273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:41 dist-test-slave-pp80 krb5kdc[4273](info): set up 2 sockets
Sep 27 22:40:41 dist-test-slave-pp80 krb5kdc[4273](info): commencing operation
krb5kdc: starting...
W20260927 22:40:42.956822 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:43 dist-test-slave-pp80 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:43.161886 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42869
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:43.485008 4286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:43.485306 4286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:43.485455 4286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:43.492753 4286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:43.492869 4286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:43.492962 4286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:43.493054 4286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:43.493131 4286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:43.503088 4286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:43.504753 4286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:43.506026 4286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:43.517879 4291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:43.517812 4294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:43.517786 4292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:43.518596 4286 server_base.cc:1034] running on GCE node
I20260927 22:40:43.519526 4286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:43.521024 4286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:43.522284 4286 hybrid_clock.cc:648] HybridClock initialized: now 1790548843522246 us; error 56 us; skew 500 ppm
I20260927 22:40:43.522647 4286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:43 dist-test-slave-pp80 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548843, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:43.532344 4286 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:43.533960 4286 webserver.cc:460] Webserver started at http://127.1.53.126:37739/ using document root <none> and password file <none>
I20260927 22:40:43.534695 4286 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:43.534853 4286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:43.535171 4286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:43.538225 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "a3e77776725a40b09f05bcdec1d377e5"
format_stamp: "Formatted at 2026-09-27 22:40:43 on dist-test-slave-pp80"
server_key: "590fd31ffc1db8a55de9c238c7ddd04c"
server_key_iv: "d45969d4eba122b9d22b2ed3b231a7f4"
server_key_version: "encryptionkey@0"
I20260927 22:40:43.539000 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "a3e77776725a40b09f05bcdec1d377e5"
format_stamp: "Formatted at 2026-09-27 22:40:43 on dist-test-slave-pp80"
server_key: "590fd31ffc1db8a55de9c238c7ddd04c"
server_key_iv: "d45969d4eba122b9d22b2ed3b231a7f4"
server_key_version: "encryptionkey@0"
I20260927 22:40:43.544391 4286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:40:43.547853 4301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:43.549245 4286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:40:43.549430 4286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "a3e77776725a40b09f05bcdec1d377e5"
format_stamp: "Formatted at 2026-09-27 22:40:43 on dist-test-slave-pp80"
server_key: "590fd31ffc1db8a55de9c238c7ddd04c"
server_key_iv: "d45969d4eba122b9d22b2ed3b231a7f4"
server_key_version: "encryptionkey@0"
I20260927 22:40:43.549630 4286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:43.575430 4286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:43.580089 4286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:43.580482 4286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:43.599958 4286 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42869
I20260927 22:40:43.599998 4352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42869 every 8 connection(s)
I20260927 22:40:43.601397 4286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:43.605468 4353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:43.611387 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4286
I20260927 22:40:43.611634 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:43.612358 1237 external_mini_cluster.cc:1428] Setting key 7325f935d637928f77c3e812edf7fa66
I20260927 22:40:43.616762 4353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5: Bootstrap starting.
I20260927 22:40:43.620599 4353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:43.621876 4353 log.cc:824] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:43.625679 4353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5: No bootstrap required, opened a new log
Sep 27 22:40:43 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:43.635403 4353 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e77776725a40b09f05bcdec1d377e5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42869 } }
I20260927 22:40:43.635794 4353 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:43.636077 4353 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e77776725a40b09f05bcdec1d377e5, State: Initialized, Role: FOLLOWER
I20260927 22:40:43.636948 4353 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e77776725a40b09f05bcdec1d377e5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42869 } }
I20260927 22:40:43.637331 4353 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:43.637517 4353 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:43.637729 4353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:43.638567 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:43.614539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.1.53.126:42869)
0927 22:40:43.615350 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:43.615367 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:43.615417 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:40:43.617709 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:43.617736 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:43.618250 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:43.618754 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:43.620048 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:43.622107 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:43.623938 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:43.625066 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:43.625289 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:43.629905 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:43.629917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:43.629934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:43.629977 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:43.633669 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:43.634590 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:43.634596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:43.634603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:43.634775 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:43.635260 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:43.635264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:43.635266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:43.635899 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:43.636233 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:43.636810 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 22:40:43.637348 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":367,"threads_started":1}
I20260927 22:40:43.640208 4353 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e77776725a40b09f05bcdec1d377e5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42869 } }
I20260927 22:40:43.640769 4353 leader_election.cc:302] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e77776725a40b09f05bcdec1d377e5; no voters:
I20260927 22:40:43.641458 4353 leader_election.cc:288] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:43.641580 4358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:43.642298 4358 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [term 1 LEADER]: Becoming Leader. State: Replica: a3e77776725a40b09f05bcdec1d377e5, State: Running, Role: LEADER
I20260927 22:40:43.642858 4358 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e77776725a40b09f05bcdec1d377e5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42869 } }
I20260927 22:40:43.643747 4353 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:43.645802 4359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e77776725a40b09f05bcdec1d377e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e77776725a40b09f05bcdec1d377e5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42869 } } }
I20260927 22:40:43.645915 4360 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e77776725a40b09f05bcdec1d377e5. Latest consensus state: current_term: 1 leader_uuid: "a3e77776725a40b09f05bcdec1d377e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e77776725a40b09f05bcdec1d377e5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42869 } } }
I20260927 22:40:43.646307 4359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:43.646427 4360 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:43.647338 4363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:43.652815 4363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:43.664412 4363 catalog_manager.cc:1347] Generated new cluster ID: c4e5a65b346e46f283d91efb6ba981e6
I20260927 22:40:43.664557 4363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:43.678830 4363 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:43.680305 4363 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:43.687574 4363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e77776725a40b09f05bcdec1d377e5: Generated new TSK 0
I20260927 22:40:43.688524 4363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:43.749361 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42869
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:44.082316 4379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:44.082574 4379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:44.082718 4379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:44.090378 4379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:44.090481 4379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:44.090647 4379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:44.100438 4379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:44.101702 4379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:44.102981 4379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:44.116392 4385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:44.116457 4384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:44.116426 4387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:44.117141 4379 server_base.cc:1034] running on GCE node
I20260927 22:40:44.117722 4379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:44.118657 4379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:44.119922 4379 hybrid_clock.cc:648] HybridClock initialized: now 1790548844119850 us; error 62 us; skew 500 ppm
I20260927 22:40:44.120339 4379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:44 dist-test-slave-pp80 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:44.123551 4379 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:44.125129 4379 webserver.cc:460] Webserver started at http://127.1.53.65:35565/ using document root <none> and password file <none>
I20260927 22:40:44.125787 4379 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:44.125906 4379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:44.126154 4379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:44.129171 4379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "01f9aad57af04b3e912266e94cbe7fec"
format_stamp: "Formatted at 2026-09-27 22:40:44 on dist-test-slave-pp80"
server_key: "8fe309d6d788b95d5ae23c4ff82947b6"
server_key_iv: "0ac2aa90c52123d34907ce9988a3df3a"
server_key_version: "encryptionkey@0"
I20260927 22:40:44.129925 4379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "01f9aad57af04b3e912266e94cbe7fec"
format_stamp: "Formatted at 2026-09-27 22:40:44 on dist-test-slave-pp80"
server_key: "8fe309d6d788b95d5ae23c4ff82947b6"
server_key_iv: "0ac2aa90c52123d34907ce9988a3df3a"
server_key_version: "encryptionkey@0"
I20260927 22:40:44.135258 4379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:40:44.139739 4394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:44.141320 4379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:44.141454 4379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "01f9aad57af04b3e912266e94cbe7fec"
format_stamp: "Formatted at 2026-09-27 22:40:44 on dist-test-slave-pp80"
server_key: "8fe309d6d788b95d5ae23c4ff82947b6"
server_key_iv: "0ac2aa90c52123d34907ce9988a3df3a"
server_key_version: "encryptionkey@0"
I20260927 22:40:44.141633 4379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:44.193769 4379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:44.198225 4379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:44.198493 4379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:44.199496 4379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:44.200912 4379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:44.200991 4379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:44.201102 4379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:44.201144 4379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:44.234354 4379 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:37675
I20260927 22:40:44.234453 4506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:37675 every 8 connection(s)
I20260927 22:40:44.235626 4379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:44.241263 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4379
I20260927 22:40:44.241479 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:44.242132 1237 external_mini_cluster.cc:1428] Setting key a5c923fcfda2937770c81665d2036d9c
Sep 27 22:40:44 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:44.258002 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:44.238315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:57975 (local address 127.1.53.126:42869)
0927 22:40:44.238792 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:44.238799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:44.239894 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 22:40:44.241825 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:44.241831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:44.241928 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:44.242048 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:44.243615 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.244518 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.246313 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.246766 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.246899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:44.250474 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:44.250479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:44.250482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:44.250513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:44.253435 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:44.254478 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:44.254486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:44.254493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:44.254746 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:44.255316 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:44.255324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:44.255330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:44.255899 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:44.256133 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:44.257217 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0927 22:40:44.257474 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260927 22:40:44.259133 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:44.238978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42869 (local address 127.1.53.65:57975)
0927 22:40:44.239740 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:44.239777 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:44.241576 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:44.242306 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:44.242316 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:44.242804 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:44.243460 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:44.243477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.244657 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.244662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:44.246075 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:44.246086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.246979 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.246986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:44.247224 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:44.247959 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:44.247984 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:44.250262 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:44.253616 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:44.253625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:44.253644 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:44.254335 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:44.254902 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:44.254908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:44.254912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:44.255188 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:44.256065 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:44.256074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:44.256660 (+ 586us) client_negotiation.cc:769] Sending connection context
0927 22:40:44.257032 (+ 372us) client_negotiation.cc:240] Negotiation successful
0927 22:40:44.257654 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":291,"threads_started":1}
I20260927 22:40:44.261061 4507 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42869
I20260927 22:40:44.261397 4507 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:40:44.262277 4507 heartbeater.cc:511] Master 127.1.53.126:42869 requested a full tablet report, sending...
I20260927 22:40:44.264576 4318 ts_manager.cc:194] Registered new tserver with Master: 01f9aad57af04b3e912266e94cbe7fec (127.1.53.65:37675)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:40:44.266073 4318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:57975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:44.288226 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42869
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:44.619124 4513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:44.619494 4513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:44.619629 4513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:44.628518 4513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:44.628628 4513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:44.628737 4513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:44.638387 4513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:44.639772 4513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:44.641423 4513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:44.655939 4518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:44.656019 4521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:44.655992 4519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:44.661067 4513 server_base.cc:1034] running on GCE node
I20260927 22:40:44.661757 4513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:44.662774 4513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:44.664040 4513 hybrid_clock.cc:648] HybridClock initialized: now 1790548844663992 us; error 55 us; skew 500 ppm
I20260927 22:40:44.664472 4513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:44 dist-test-slave-pp80 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:44.667763 4513 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:44.669402 4513 webserver.cc:460] Webserver started at http://127.1.53.66:39081/ using document root <none> and password file <none>
I20260927 22:40:44.670140 4513 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:44.670279 4513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:44.670584 4513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:44.673691 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "baa30224e33d4564ad815a7331573c32"
format_stamp: "Formatted at 2026-09-27 22:40:44 on dist-test-slave-pp80"
server_key: "44f14c45ca33ec11d73b50ece2a28e7d"
server_key_iv: "9ee729249b414b23c0075b37157f464f"
server_key_version: "encryptionkey@0"
I20260927 22:40:44.674535 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "baa30224e33d4564ad815a7331573c32"
format_stamp: "Formatted at 2026-09-27 22:40:44 on dist-test-slave-pp80"
server_key: "44f14c45ca33ec11d73b50ece2a28e7d"
server_key_iv: "9ee729249b414b23c0075b37157f464f"
server_key_version: "encryptionkey@0"
I20260927 22:40:44.679913 4513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:40:44.683432 4528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:44.684895 4513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:44.685070 4513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "baa30224e33d4564ad815a7331573c32"
format_stamp: "Formatted at 2026-09-27 22:40:44 on dist-test-slave-pp80"
server_key: "44f14c45ca33ec11d73b50ece2a28e7d"
server_key_iv: "9ee729249b414b23c0075b37157f464f"
server_key_version: "encryptionkey@0"
I20260927 22:40:44.685276 4513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:44.709429 4513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:44.713923 4513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:44.714285 4513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:44.715301 4513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:44.716686 4513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:44.716758 4513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:44.716840 4513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:44.716884 4513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:44.749955 4513 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:37275
I20260927 22:40:44.750079 4640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:37275 every 8 connection(s)
I20260927 22:40:44.751264 4513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:44.757412 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4513
I20260927 22:40:44.757615 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:44.758158 1237 external_mini_cluster.cc:1428] Setting key 6edb666fe019c63bfd117ac6c888a457
Sep 27 22:40:44 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548844, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:44.772792 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:44.754021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:57515 (local address 127.1.53.126:42869)
0927 22:40:44.754199 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:44.754204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:44.755524 (+ 1320us) server_negotiation.cc:407] Connection header received
0927 22:40:44.757602 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:44.757609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:44.757718 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:44.757805 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:44.759762 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.760790 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.762385 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.763110 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.763254 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:44.766975 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:44.766980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:44.766983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:44.767021 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:44.769608 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:44.770367 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:44.770377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:44.770380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:44.770501 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:44.770930 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:44.770935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:44.770939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:44.771218 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:44.771382 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:44.772168 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 22:40:44.772432 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:40:44.773878 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:44.754584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42869 (local address 127.1.53.66:57515)
0927 22:40:44.755348 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:44.755384 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:44.757208 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:44.758091 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:44.758103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:44.758844 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:44.759554 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:44.759579 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.760963 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.760967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:44.762236 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:44.762248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:44.763325 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:44.763329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:44.763536 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:44.764336 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:44.764360 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:44.766729 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:44.769768 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:44.769778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:44.769796 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:44.770238 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:44.770608 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:44.770615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:44.770618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:44.770811 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:44.771362 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:44.771370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:44.771732 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 22:40:44.772087 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 22:40:44.772714 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":275,"threads_started":1}
I20260927 22:40:44.775470 4641 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42869
I20260927 22:40:44.775847 4641 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
I20260927 22:40:44.776845 4641 heartbeater.cc:511] Master 127.1.53.126:42869 requested a full tablet report, sending...
I20260927 22:40:44.778223 4318 ts_manager.cc:194] Registered new tserver with Master: baa30224e33d4564ad815a7331573c32 (127.1.53.66:37275)
I20260927 22:40:44.779023 4318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:57515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:44.803700 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42869
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:45.124090 4646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:45.124444 4646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:45.124596 4646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:45.131812 4646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:45.131925 4646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:45.132108 4646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:45.143105 4646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:45.144671 4646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:45.146284 4646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:45.164685 4651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:45.164625 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:45.164718 4652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:45.165796 4646 server_base.cc:1034] running on GCE node
I20260927 22:40:45.166491 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:45.167586 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:45.168819 4646 hybrid_clock.cc:648] HybridClock initialized: now 1790548845168757 us; error 58 us; skew 500 ppm
I20260927 22:40:45.169191 4646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:45.172785 4646 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:45.174412 4646 webserver.cc:460] Webserver started at http://127.1.53.67:35959/ using document root <none> and password file <none>
I20260927 22:40:45.175182 4646 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:45.175325 4646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:45.175648 4646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:45.178697 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "71fc94c365cd45ab832b1d56af11dad1"
format_stamp: "Formatted at 2026-09-27 22:40:45 on dist-test-slave-pp80"
server_key: "7a4d459fab3b9bd47602ce9c6a649a67"
server_key_iv: "d48e789f96960d6bffdf988288f8c168"
server_key_version: "encryptionkey@0"
I20260927 22:40:45.179495 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "71fc94c365cd45ab832b1d56af11dad1"
format_stamp: "Formatted at 2026-09-27 22:40:45 on dist-test-slave-pp80"
server_key: "7a4d459fab3b9bd47602ce9c6a649a67"
server_key_iv: "d48e789f96960d6bffdf988288f8c168"
server_key_version: "encryptionkey@0"
I20260927 22:40:45.184953 4646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:40:45.188634 4661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:45.190169 4646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:40:45.190341 4646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "71fc94c365cd45ab832b1d56af11dad1"
format_stamp: "Formatted at 2026-09-27 22:40:45 on dist-test-slave-pp80"
server_key: "7a4d459fab3b9bd47602ce9c6a649a67"
server_key_iv: "d48e789f96960d6bffdf988288f8c168"
server_key_version: "encryptionkey@0"
I20260927 22:40:45.190544 4646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:45.218567 4646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:45.222918 4646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:45.223232 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:45.224255 4646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:45.225617 4646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:45.225716 4646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:45.225870 4646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:45.225941 4646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:45.260675 4646 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:38311
I20260927 22:40:45.260761 4773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:38311 every 8 connection(s)
I20260927 22:40:45.262004 4646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:45.268889 4507 heartbeater.cc:503] Master 127.1.53.126:42869 was elected leader, sending a full tablet report...
I20260927 22:40:45.272480 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4646
I20260927 22:40:45.272657 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:45.273113 1237 external_mini_cluster.cc:1428] Setting key 50676fb58111b1fe5c28e4b6404eb04d
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:45.283059 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:45.264777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:60683 (local address 127.1.53.126:42869)
0927 22:40:45.264983 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:45.264990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:45.266321 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 22:40:45.268135 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:45.268140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:45.268301 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:45.268405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:45.270208 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.271163 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.273034 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.273476 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.273607 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:45.277538 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:45.277543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:45.277547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:45.277580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:45.280326 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.281032 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.281037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.281040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.281171 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.281551 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.281555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.281557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.281832 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:45.281943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:45.282577 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 22:40:45.282756 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:40:45.284041 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:45.265382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42869 (local address 127.1.53.67:60683)
0927 22:40:45.266142 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:45.266185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:45.267841 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:45.268626 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:45.268635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:45.269273 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:45.269986 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:45.270004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.271345 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.271349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:45.272889 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:45.272901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.273712 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.273719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:45.273951 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:45.274730 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:45.274764 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:45.277328 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:45.280456 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:45.280463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:45.280477 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:45.280913 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:45.281267 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:45.281270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:45.281272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:45.281446 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:45.281931 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:45.281936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:45.282253 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 22:40:45.282476 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 22:40:45.282875 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":285,"threads_started":1}
I20260927 22:40:45.285425 4774 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42869
I20260927 22:40:45.285706 4774 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:45.286392 4774 heartbeater.cc:511] Master 127.1.53.126:42869 requested a full tablet report, sending...
I20260927 22:40:45.288142 4318 ts_manager.cc:194] Registered new tserver with Master: 71fc94c365cd45ab832b1d56af11dad1 (127.1.53.67:38311)
I20260927 22:40:45.289058 4318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:60683
I20260927 22:40:45.301131 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:45.349470 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:45.334742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.1.53.126:42869)
0927 22:40:45.334931 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:45.334936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:45.335483 (+ 547us) server_negotiation.cc:407] Connection header received
0927 22:40:45.335663 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:45.335690 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:45.335780 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:45.335899 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:45.337302 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.338098 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.339528 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.339928 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.340033 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:45.343872 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:45.343879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:45.343883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:45.343919 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:45.346860 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.347479 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.347483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.347486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.347625 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.348044 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.348048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.348051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.348458 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:45.348687 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:45.349056 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 22:40:45.349230 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:40:45.350075 4318 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45036
I20260927 22:40:45.351816 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4379
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:45.367101 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:45.352941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.1.53.65:37675)
0927 22:40:45.353576 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:45.353583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:45.353607 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:40:45.353657 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:45.353664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:45.353925 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:45.354092 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:45.355315 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.356360 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.357487 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.358048 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.358146 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:45.361850 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:45.361860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:45.361875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:45.361907 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:45.364750 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.365324 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.365331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.365333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.365522 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.365826 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.365830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.365831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.366143 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:45.366368 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:45.366544 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 22:40:45.366720 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":329,"threads_started":1}
W20260927 22:40:45.368362 4502 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43606
W20260927 22:40:45.369467 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4379: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:45.369655 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4379
I20260927 22:40:45.385393 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4513
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:45.402349 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:45.386338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.1.53.66:37275)
0927 22:40:45.386917 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:45.386924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:45.386950 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:40:45.387017 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:45.387023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:45.387240 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:45.387413 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:45.388751 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.389838 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.391087 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.391657 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.391832 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:45.396569 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:45.396583 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:45.396599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:45.396648 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:45.399465 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.400203 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.400210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.400212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.400444 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.400773 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.400777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.400779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.401156 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:45.401461 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:45.401726 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:40:45.401939 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":308,"threads_started":1}
W20260927 22:40:45.403645 4636 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52396
W20260927 22:40:45.404803 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4513: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:45.404961 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4513
I20260927 22:40:45.420112 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4646
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548845, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:45.437688 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:45.421134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.1.53.67:38311)
0927 22:40:45.421757 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:45.421765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:45.421794 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:40:45.421852 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:45.421858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:45.422067 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:45.422252 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:45.423610 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.424718 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.425996 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:45.426565 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:45.426700 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:45.431323 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:45.431336 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:45.431352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:45.431391 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:45.434865 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.435549 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.435557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.435559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.435674 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:45.436031 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:45.436038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:45.436041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:45.436440 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:45.436743 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:45.437059 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 22:40:45.437330 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":320,"threads_started":1}
W20260927 22:40:45.439047 4769 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54224
W20260927 22:40:45.440152 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:40:45.440375 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4646
I20260927 22:40:45.456146 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4286
I20260927 22:40:45.457382 4348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:45.564183 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4286
2026-09-27T22:40:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3776 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4791](info): set up 2 sockets
Sep 27 22:40:45 dist-test-slave-pp80 krb5kdc[4791](info): commencing operation
krb5kdc: starting...
W20260927 22:40:46.672644 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:46 dist-test-slave-pp80 krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:46.877116 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33235
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:47.205458 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:47.205870 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:47.206036 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:47.213330 4804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:47.213487 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:47.213587 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:47.213686 4804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:47.213768 4804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:47.223886 4804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:33235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:47.225381 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:47.226732 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:47.238696 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:47.238673 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:47.238621 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:47.241751 4804 server_base.cc:1034] running on GCE node
I20260927 22:40:47.242637 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:47.244184 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:47.245486 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790548847245435 us; error 42 us; skew 500 ppm
I20260927 22:40:47.245903 4804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:47 dist-test-slave-pp80 krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:47.249102 4804 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:47.250702 4804 webserver.cc:460] Webserver started at http://127.1.53.126:40289/ using document root <none> and password file <none>
I20260927 22:40:47.251454 4804 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:47.251596 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:47.251901 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:47.254966 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "a44ce47ea7d9404f9b943dc006967ac7"
format_stamp: "Formatted at 2026-09-27 22:40:47 on dist-test-slave-pp80"
server_key: "96ca75f4376ddd06f59065e5944c25cf"
server_key_iv: "b3950bc738968774ef8060a1a006c3d0"
server_key_version: "encryptionkey@0"
I20260927 22:40:47.255820 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "a44ce47ea7d9404f9b943dc006967ac7"
format_stamp: "Formatted at 2026-09-27 22:40:47 on dist-test-slave-pp80"
server_key: "96ca75f4376ddd06f59065e5944c25cf"
server_key_iv: "b3950bc738968774ef8060a1a006c3d0"
server_key_version: "encryptionkey@0"
I20260927 22:40:47.261163 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:40:47.265035 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:47.266505 4804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:47.266696 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "a44ce47ea7d9404f9b943dc006967ac7"
format_stamp: "Formatted at 2026-09-27 22:40:47 on dist-test-slave-pp80"
server_key: "96ca75f4376ddd06f59065e5944c25cf"
server_key_iv: "b3950bc738968774ef8060a1a006c3d0"
server_key_version: "encryptionkey@0"
I20260927 22:40:47.266906 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:47.303121 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:47.307858 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:47.308161 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:47.327005 4804 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:33235
I20260927 22:40:47.327091 4870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:33235 every 8 connection(s)
I20260927 22:40:47.328431 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:47.332419 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:47.336889 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4804
I20260927 22:40:47.337146 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:47.338001 1237 external_mini_cluster.cc:1428] Setting key bce05fde1d47f72cdfba4fcfbe660fe5
I20260927 22:40:47.345363 4871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7: Bootstrap starting.
I20260927 22:40:47.349713 4871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:47.350867 4871 log.cc:824] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:47.354314 4871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7: No bootstrap required, opened a new log
Sep 27 22:40:47 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:47.362028 4871 raft_consensus.cc:348] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44ce47ea7d9404f9b943dc006967ac7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33235 } }
I20260927 22:40:47.362411 4871 raft_consensus.cc:374] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:47.362519 4871 raft_consensus.cc:729] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44ce47ea7d9404f9b943dc006967ac7, State: Initialized, Role: FOLLOWER
I20260927 22:40:47.362994 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:47.340169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.1.53.126:33235)
0927 22:40:47.341051 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:47.341062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:47.341105 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:40:47.343716 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:47.343742 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:47.344190 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:47.344596 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:47.346583 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:47.347942 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:47.349966 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:47.350699 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:47.350837 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:47.355432 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:47.355441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:47.355455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:47.355489 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:47.358679 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:47.359395 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:47.359402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:47.359411 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:47.359574 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:47.359966 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:47.359975 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:47.359977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:47.360603 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:47.360918 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:47.361348 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 22:40:47.361756 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":402,"threads_started":1}
I20260927 22:40:47.363109 4871 consensus_queue.cc:260] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44ce47ea7d9404f9b943dc006967ac7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33235 } }
I20260927 22:40:47.363389 4871 raft_consensus.cc:388] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:47.363557 4871 raft_consensus.cc:482] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:47.363749 4871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:47.366205 4871 raft_consensus.cc:504] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44ce47ea7d9404f9b943dc006967ac7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33235 } }
I20260927 22:40:47.366701 4871 leader_election.cc:302] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a44ce47ea7d9404f9b943dc006967ac7; no voters:
I20260927 22:40:47.367251 4871 leader_election.cc:288] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:47.367388 4876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:47.368039 4876 raft_consensus.cc:686] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [term 1 LEADER]: Becoming Leader. State: Replica: a44ce47ea7d9404f9b943dc006967ac7, State: Running, Role: LEADER
I20260927 22:40:47.368597 4876 consensus_queue.cc:237] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44ce47ea7d9404f9b943dc006967ac7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33235 } }
I20260927 22:40:47.369309 4871 sys_catalog.cc:564] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:47.371647 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a44ce47ea7d9404f9b943dc006967ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44ce47ea7d9404f9b943dc006967ac7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33235 } } }
I20260927 22:40:47.371984 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:47.372429 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a44ce47ea7d9404f9b943dc006967ac7. Latest consensus state: current_term: 1 leader_uuid: "a44ce47ea7d9404f9b943dc006967ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44ce47ea7d9404f9b943dc006967ac7" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33235 } } }
I20260927 22:40:47.374178 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:47.378214 4881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:47.385176 4881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:47.394506 4881 catalog_manager.cc:1347] Generated new cluster ID: 3268fc48ad164eb9aa19ee7f48d78c11
I20260927 22:40:47.394639 4881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:47.412577 4881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:47.413916 4881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:47.420094 4881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a44ce47ea7d9404f9b943dc006967ac7: Generated new TSK 0
I20260927 22:40:47.421258 4881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:47.469532 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33235
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:47.805339 4897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:47.805671 4897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:47.805838 4897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:47.813241 4897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:47.813396 4897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:47.813591 4897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:47.823540 4897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:47.824954 4897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:47.826469 4897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:47.840168 4903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:47.840260 4902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:47.840255 4905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:47.840965 4897 server_base.cc:1034] running on GCE node
I20260927 22:40:47.841640 4897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:47.842621 4897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:47.843868 4897 hybrid_clock.cc:648] HybridClock initialized: now 1790548847843799 us; error 63 us; skew 500 ppm
I20260927 22:40:47.844213 4897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:47 dist-test-slave-pp80 krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:47.847913 4897 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:47.849565 4897 webserver.cc:460] Webserver started at http://127.1.53.65:33039/ using document root <none> and password file <none>
I20260927 22:40:47.850267 4897 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:47.850390 4897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:47.850708 4897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:47.853760 4897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "60833c4abc9942cb9f64c97a69b543d4"
format_stamp: "Formatted at 2026-09-27 22:40:47 on dist-test-slave-pp80"
server_key: "3cdeb3f5a357a0adf2573b163624fa2b"
server_key_iv: "8ad9c78de160bba0a448aa77604c8598"
server_key_version: "encryptionkey@0"
I20260927 22:40:47.854539 4897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "60833c4abc9942cb9f64c97a69b543d4"
format_stamp: "Formatted at 2026-09-27 22:40:47 on dist-test-slave-pp80"
server_key: "3cdeb3f5a357a0adf2573b163624fa2b"
server_key_iv: "8ad9c78de160bba0a448aa77604c8598"
server_key_version: "encryptionkey@0"
I20260927 22:40:47.859738 4897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:40:47.863468 4912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:47.865032 4897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:47.865226 4897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "60833c4abc9942cb9f64c97a69b543d4"
format_stamp: "Formatted at 2026-09-27 22:40:47 on dist-test-slave-pp80"
server_key: "3cdeb3f5a357a0adf2573b163624fa2b"
server_key_iv: "8ad9c78de160bba0a448aa77604c8598"
server_key_version: "encryptionkey@0"
I20260927 22:40:47.865444 4897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:47.879033 4897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:47.883625 4897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:47.883942 4897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:47.884974 4897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:47.886380 4897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:47.886479 4897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:47.886632 4897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:47.886701 4897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:47.920814 4897 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:34211
I20260927 22:40:47.920898 5024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:34211 every 8 connection(s)
I20260927 22:40:47.922092 4897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:47.929849 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 4897
I20260927 22:40:47.930068 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:47.930662 1237 external_mini_cluster.cc:1428] Setting key 16f499df897d8a87d87d113c1c0ed001
Sep 27 22:40:47 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:47.942816 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:47.924732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:35747 (local address 127.1.53.126:33235)
0927 22:40:47.925200 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:47.925205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:47.926167 (+ 962us) server_negotiation.cc:407] Connection header received
0927 22:40:47.928115 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:47.928121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:47.928212 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:47.928371 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:47.929922 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:47.930807 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:47.932735 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:47.933172 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:47.933310 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:47.936908 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:47.936913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:47.936916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:47.936948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:47.939600 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:47.940404 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:47.940412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:47.940416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:47.940549 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:47.941001 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:47.941006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:47.941010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:47.941281 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:47.941444 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:47.942184 (+ 740us) server_negotiation.cc:299] Negotiation successful
0927 22:40:47.942416 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":273,"threads_started":1}
I20260927 22:40:47.943822 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:47.925272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33235 (local address 127.1.53.65:35747)
0927 22:40:47.925986 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:47.926053 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:47.927821 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:47.928609 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:47.928618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:47.929149 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:47.929748 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:47.929766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:47.930963 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:47.930969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:47.932590 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:47.932606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:47.933402 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:47.933412 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:47.933671 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:47.934382 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:47.934404 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:47.936679 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:47.939763 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:47.939772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:47.939790 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:47.940281 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:47.940666 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:47.940673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:47.940677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:47.940874 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:47.941418 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:47.941427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:47.941769 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 22:40:47.942060 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 22:40:47.942612 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":278,"threads_started":1}
I20260927 22:40:47.945725 5025 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33235
I20260927 22:40:47.946070 5025 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:47.946841 5025 heartbeater.cc:511] Master 127.1.53.126:33235 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:40:47.949170 4836 ts_manager.cc:194] Registered new tserver with Master: 60833c4abc9942cb9f64c97a69b543d4 (127.1.53.65:34211)
I20260927 22:40:47.950611 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:35747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:47.975762 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33235
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:48.301882 5031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:48.302161 5031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:48.302254 5031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:48.309628 5031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:48.309749 5031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:48.309864 5031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:48.319669 5031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:48.320927 5031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:48.322177 5031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:48.340802 5036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:48.341045 5037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:48.341595 5039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:48.343065 5031 server_base.cc:1034] running on GCE node
I20260927 22:40:48.343674 5031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:48.344648 5031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:48.345911 5031 hybrid_clock.cc:648] HybridClock initialized: now 1790548848345793 us; error 58 us; skew 500 ppm
I20260927 22:40:48.346261 5031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:48 dist-test-slave-pp80 krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:48.349753 5031 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:48.351410 5031 webserver.cc:460] Webserver started at http://127.1.53.66:44509/ using document root <none> and password file <none>
I20260927 22:40:48.352310 5031 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:48.352460 5031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:48.352807 5031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:48.356029 5031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "be11d32294a14c29a6d2b5d72beb644b"
format_stamp: "Formatted at 2026-09-27 22:40:48 on dist-test-slave-pp80"
server_key: "15afdc773c8e06aff61d9f4189c8386e"
server_key_iv: "da7523f9b90cfc58ebc6d633117aee01"
server_key_version: "encryptionkey@0"
I20260927 22:40:48.356902 5031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "be11d32294a14c29a6d2b5d72beb644b"
format_stamp: "Formatted at 2026-09-27 22:40:48 on dist-test-slave-pp80"
server_key: "15afdc773c8e06aff61d9f4189c8386e"
server_key_iv: "da7523f9b90cfc58ebc6d633117aee01"
server_key_version: "encryptionkey@0"
I20260927 22:40:48.362277 5031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:48.365933 5046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:48.367244 5031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:40:48.367416 5031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "be11d32294a14c29a6d2b5d72beb644b"
format_stamp: "Formatted at 2026-09-27 22:40:48 on dist-test-slave-pp80"
server_key: "15afdc773c8e06aff61d9f4189c8386e"
server_key_iv: "da7523f9b90cfc58ebc6d633117aee01"
server_key_version: "encryptionkey@0"
I20260927 22:40:48.367615 5031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:48.384775 5031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:48.388629 5031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:48.388953 5031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:48.389928 5031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:48.391283 5031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:48.391382 5031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:48.391532 5031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:48.391601 5031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:48.425262 5031 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:41347
I20260927 22:40:48.425309 5158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:41347 every 8 connection(s)
I20260927 22:40:48.426539 5031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:48.435243 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5031
I20260927 22:40:48.435441 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:48.435830 1237 external_mini_cluster.cc:1428] Setting key 3f85f65d16a42c85dc37b56ba3e21244
Sep 27 22:40:48 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:48.448400 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:48.429157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:51207 (local address 127.1.53.126:33235)
0927 22:40:48.429370 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:48.429376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:48.430640 (+ 1264us) server_negotiation.cc:407] Connection header received
0927 22:40:48.432511 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:48.432516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:48.432585 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:48.432690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:48.434194 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.435008 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.437057 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.437533 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.437674 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:48.442187 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:48.442192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:48.442195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:48.442226 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:48.445141 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:48.445842 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:48.445850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:48.445857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:48.446025 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:48.446411 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:48.446419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:48.446425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:48.446791 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:48.446952 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:48.447657 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 22:40:48.447908 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:40:48.448952 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:48.429655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33235 (local address 127.1.53.66:51207)
0927 22:40:48.430442 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:48.430487 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:48.432155 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:48.432888 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:48.432897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:48.433455 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:48.434026 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:48.434043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.435215 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.435223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:48.436896 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:48.436908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.437739 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.437744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:48.438110 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:48.438831 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:48.438870 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:48.441917 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:48.445271 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:48.445278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:48.445292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:48.445706 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:48.446128 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:48.446131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:48.446134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:48.446299 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:48.446905 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:48.446911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:48.447254 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 22:40:48.447516 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:40:48.447886 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":333,"threads_started":1}
I20260927 22:40:48.450377 5159 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33235
I20260927 22:40:48.450640 5159 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:48.451313 5159 heartbeater.cc:511] Master 127.1.53.126:33235 requested a full tablet report, sending...
I20260927 22:40:48.452781 4836 ts_manager.cc:194] Registered new tserver with Master: be11d32294a14c29a6d2b5d72beb644b (127.1.53.66:41347)
I20260927 22:40:48.453578 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:51207
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:48.483911 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33235
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:48.804661 5164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:48.805042 5164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:48.805222 5164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:48.812577 5164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:48.812732 5164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:48.812915 5164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:48.822532 5164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:48.823849 5164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:48.825280 5164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:48.838722 5169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:48.838744 5172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:48.838645 5170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:48.839449 5164 server_base.cc:1034] running on GCE node
I20260927 22:40:48.840085 5164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:48.841092 5164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:48.842312 5164 hybrid_clock.cc:648] HybridClock initialized: now 1790548848842260 us; error 42 us; skew 500 ppm
I20260927 22:40:48.842681 5164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:48 dist-test-slave-pp80 krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:48.853711 5164 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:48.855526 5164 webserver.cc:460] Webserver started at http://127.1.53.67:37901/ using document root <none> and password file <none>
I20260927 22:40:48.856359 5164 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:48.856508 5164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:48.856832 5164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:48.860464 5164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "c603ca09808d48aabaf64b2f10a23dc6"
format_stamp: "Formatted at 2026-09-27 22:40:48 on dist-test-slave-pp80"
server_key: "08cd236e8947194e8904d3cf4ec31070"
server_key_iv: "934d520e7fd5abbba62208a6266a2cf1"
server_key_version: "encryptionkey@0"
I20260927 22:40:48.861253 5164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "c603ca09808d48aabaf64b2f10a23dc6"
format_stamp: "Formatted at 2026-09-27 22:40:48 on dist-test-slave-pp80"
server_key: "08cd236e8947194e8904d3cf4ec31070"
server_key_iv: "934d520e7fd5abbba62208a6266a2cf1"
server_key_version: "encryptionkey@0"
I20260927 22:40:48.866536 5164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:40:48.870193 5179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:48.871912 5164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:40:48.872133 5164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "c603ca09808d48aabaf64b2f10a23dc6"
format_stamp: "Formatted at 2026-09-27 22:40:48 on dist-test-slave-pp80"
server_key: "08cd236e8947194e8904d3cf4ec31070"
server_key_iv: "934d520e7fd5abbba62208a6266a2cf1"
server_key_version: "encryptionkey@0"
I20260927 22:40:48.872524 5164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:48.891461 5164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:48.895309 5164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:48.895588 5164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:48.896509 5164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:48.897785 5164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:48.897857 5164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:48.898003 5164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:48.898053 5164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:48.931356 5164 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:37323
I20260927 22:40:48.931442 5291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:37323 every 8 connection(s)
I20260927 22:40:48.932667 5164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:48.942399 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5164
I20260927 22:40:48.942595 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:48.943113 1237 external_mini_cluster.cc:1428] Setting key 22e70944a36d3364a32ef9e564e93a5a
Sep 27 22:40:48 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:48.952886 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:48.935350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:43355 (local address 127.1.53.126:33235)
0927 22:40:48.935526 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:48.935531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:48.936629 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 22:40:48.938497 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:48.938501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:48.938595 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:48.938683 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:48.940309 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.941085 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.942793 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.943429 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.943566 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:48.947539 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:48.947546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:48.947551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:48.947589 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:48.949922 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:48.950635 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:48.950643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:48.950648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:48.950774 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:48.951181 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:48.951186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:48.951189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:48.951452 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:48.951605 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:48.952300 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 22:40:48.952515 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:40:48.953805 5025 heartbeater.cc:503] Master 127.1.53.126:33235 was elected leader, sending a full tablet report...
I20260927 22:40:48.953915 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:48.935696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33235 (local address 127.1.53.67:43355)
0927 22:40:48.936459 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:48.936505 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:48.938237 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:48.938955 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:48.938964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:48.939487 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:48.940090 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:48.940113 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.941241 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.941245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:48.942632 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:48.942644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:48.943690 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:48.943698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:48.943982 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:48.944817 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:48.944867 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:48.947347 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:48.950083 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:48.950092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:48.950108 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:48.950521 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:48.950880 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:48.950886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:48.950889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:48.951068 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:48.951578 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:48.951586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:48.951917 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 22:40:48.952189 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 22:40:48.952759 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":290,"threads_started":1}
I20260927 22:40:48.955494 5292 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33235
I20260927 22:40:48.955782 5292 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:48.956537 5292 heartbeater.cc:511] Master 127.1.53.126:33235 requested a full tablet report, sending...
I20260927 22:40:48.958086 4836 ts_manager.cc:194] Registered new tserver with Master: c603ca09808d48aabaf64b2f10a23dc6 (127.1.53.67:37323)
I20260927 22:40:48.958894 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:43355
I20260927 22:40:48.970750 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:40:48 dist-test-slave-pp80 krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:49.020418 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:49.004708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.1.53.65:34211)
0927 22:40:49.005429 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:49.005440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:49.005545 (+ 105us) server_negotiation.cc:407] Connection header received
0927 22:40:49.005792 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:49.005804 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:49.006056 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:49.006227 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:49.007632 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.008717 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.010057 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.010622 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.010760 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:49.014794 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:49.014805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:49.014821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:49.014854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:49.017690 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.018311 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.018318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.018320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.018436 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.018803 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.018809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.018813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.019154 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:49.019448 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:49.019742 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 22:40:49.020002 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":363,"threads_started":1}
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:49.038931 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:49.024678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.1.53.126:33235)
0927 22:40:49.024904 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:49.024909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:49.025040 (+ 131us) server_negotiation.cc:407] Connection header received
0927 22:40:49.025222 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:49.025227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:49.025297 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:49.025400 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:49.026578 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.027347 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.028649 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.029132 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.029234 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:49.033343 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:49.033349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:49.033352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:49.033404 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:49.036202 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.037022 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.037027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.037029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.037153 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.037574 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.037582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.037585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.037965 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:49.038171 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:49.038496 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:40:49.038670 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 22:40:49.039572 4836 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51082
I20260927 22:40:49.040621 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4897
I20260927 22:40:49.052726 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:49.041566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.1.53.65:34211)
0927 22:40:49.041778 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:49.041784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:49.041878 (+ 94us) server_negotiation.cc:407] Connection header received
0927 22:40:49.042009 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:49.042013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:49.042071 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:49.042152 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:49.043255 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.043958 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.045159 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.045618 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.045727 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:49.047849 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:49.047855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:49.047859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:49.047894 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:49.050242 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.050779 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.050783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.050785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.050909 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.051346 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.051350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.051352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.051625 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:49.051825 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:49.052162 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:40:49.052456 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:40:49.053391 5020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:49.135954 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4897
I20260927 22:40:49.152299 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5031
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:49.168125 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:49.153203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.1.53.66:41347)
0927 22:40:49.153881 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:49.153889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:49.153919 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:40:49.153999 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:49.154006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:49.154278 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:49.154435 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:49.155683 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.156891 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.158159 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.158739 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.158897 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:49.162784 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:49.162795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:49.162810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:49.162843 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:49.165727 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.166335 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.166342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.166344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.166456 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.166758 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.166761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.166763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.167089 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:49.167301 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:49.167519 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 22:40:49.167699 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":350,"threads_started":1}
I20260927 22:40:49.169421 5154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:49.251787 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5031
I20260927 22:40:49.267889 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5164
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:49.285888 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:49.268962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.1.53.67:37323)
0927 22:40:49.269634 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:49.269642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:49.269674 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:40:49.269801 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:49.269815 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:49.270106 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:49.270307 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:49.271765 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.273011 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.274461 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:49.275121 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:49.275278 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:49.280150 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:49.280161 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:49.280177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:49.280212 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:49.283120 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.283782 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.283789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.283792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.283922 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:49.284309 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:49.284317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:49.284320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:49.284663 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:49.284982 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:49.285277 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 22:40:49.285548 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":374,"threads_started":1}
I20260927 22:40:49.287279 5287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:49.371265 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5164
I20260927 22:40:49.386984 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 4804
I20260927 22:40:49.388136 4866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:49.492045 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 4804
2026-09-27T22:40:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3928 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[5311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[5311](info): set up 2 sockets
Sep 27 22:40:49 dist-test-slave-pp80 krb5kdc[5311](info): commencing operation
krb5kdc: starting...
W20260927 22:40:50.592641 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:40:50 dist-test-slave-pp80 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:40:50.795826 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:34021
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:34021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:51.119887 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:51.120242 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:51.120424 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:51.127691 5324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:51.127857 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:51.127964 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:51.128062 5324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:51.128149 5324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:51.138257 5324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:34021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:51.139611 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:51.141000 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:51.152398 5330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:51.152523 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:51.152557 5332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:51.153268 5324 server_base.cc:1034] running on GCE node
I20260927 22:40:51.154104 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:51.155509 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:51.156706 5324 hybrid_clock.cc:648] HybridClock initialized: now 1790548851156666 us; error 29 us; skew 500 ppm
I20260927 22:40:51.157083 5324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:51 dist-test-slave-pp80 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:51.160135 5324 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:40:51.161222 5324 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:51.161350 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:51.161664 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:51.164728 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "ee8a0cec76fa467f807cda3c16902d05"
format_stamp: "Formatted at 2026-09-27 22:40:51 on dist-test-slave-pp80"
server_key: "d3047082cd49758a0bfa3b97c012a7d3"
server_key_iv: "59146b99ffcaf7f02bf2e5a9b842d777"
server_key_version: "encryptionkey@0"
I20260927 22:40:51.165519 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "ee8a0cec76fa467f807cda3c16902d05"
format_stamp: "Formatted at 2026-09-27 22:40:51 on dist-test-slave-pp80"
server_key: "d3047082cd49758a0bfa3b97c012a7d3"
server_key_iv: "59146b99ffcaf7f02bf2e5a9b842d777"
server_key_version: "encryptionkey@0"
I20260927 22:40:51.170698 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:40:51.174152 5338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:51.175661 5324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:40:51.175859 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "ee8a0cec76fa467f807cda3c16902d05"
format_stamp: "Formatted at 2026-09-27 22:40:51 on dist-test-slave-pp80"
server_key: "d3047082cd49758a0bfa3b97c012a7d3"
server_key_iv: "59146b99ffcaf7f02bf2e5a9b842d777"
server_key_version: "encryptionkey@0"
I20260927 22:40:51.176087 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:51.193140 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:51.197402 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:51.197683 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:51.216634 5324 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:34021
I20260927 22:40:51.216673 5389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:34021 every 8 connection(s)
I20260927 22:40:51.217572 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:51.221349 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:51.225708 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5324
I20260927 22:40:51.225941 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:51.226541 1237 external_mini_cluster.cc:1428] Setting key f92e5aa8e7635fa021d011bdea388df9
I20260927 22:40:51.232142 5390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05: Bootstrap starting.
I20260927 22:40:51.235950 5390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:51.237207 5390 log.cc:824] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:51.240556 5390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05: No bootstrap required, opened a new log
Sep 27 22:40:51 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:51.251470 5390 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8a0cec76fa467f807cda3c16902d05" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34021 } }
I20260927 22:40:51.251873 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:51.228502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.1.53.126:34021)
0927 22:40:51.229303 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:51.229318 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:51.229363 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:40:51.231060 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:51.231083 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:51.231454 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:51.231775 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:51.233675 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:51.235015 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:51.236898 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:51.237644 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:51.237838 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:51.242353 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:51.242363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:51.242387 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:51.242441 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:51.246599 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:51.247739 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:51.247748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:51.247757 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:51.247970 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:51.248433 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:51.248438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:51.248441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:51.249239 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:51.249530 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:51.249975 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 22:40:51.250475 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":386,"threads_started":1}
I20260927 22:40:51.251946 5390 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:51.252048 5390 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8a0cec76fa467f807cda3c16902d05, State: Initialized, Role: FOLLOWER
I20260927 22:40:51.252761 5390 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8a0cec76fa467f807cda3c16902d05" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34021 } }
I20260927 22:40:51.253024 5390 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:51.253214 5390 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:51.253392 5390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:51.256127 5390 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8a0cec76fa467f807cda3c16902d05" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34021 } }
I20260927 22:40:51.256644 5390 leader_election.cc:302] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8a0cec76fa467f807cda3c16902d05; no voters:
I20260927 22:40:51.257328 5390 leader_election.cc:288] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:51.257531 5395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:51.258809 5395 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [term 1 LEADER]: Becoming Leader. State: Replica: ee8a0cec76fa467f807cda3c16902d05, State: Running, Role: LEADER
I20260927 22:40:51.259325 5395 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8a0cec76fa467f807cda3c16902d05" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34021 } }
I20260927 22:40:51.259588 5390 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:51.262563 5397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8a0cec76fa467f807cda3c16902d05. Latest consensus state: current_term: 1 leader_uuid: "ee8a0cec76fa467f807cda3c16902d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8a0cec76fa467f807cda3c16902d05" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34021 } } }
I20260927 22:40:51.262892 5397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:51.263515 5396 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee8a0cec76fa467f807cda3c16902d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8a0cec76fa467f807cda3c16902d05" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34021 } } }
I20260927 22:40:51.263832 5396 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:51.263836 5401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:51.271706 5401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:51.281450 5401 catalog_manager.cc:1347] Generated new cluster ID: ebab7b0d89ab4c11a6b082ce647a5801
I20260927 22:40:51.281590 5401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:51.297891 5401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:51.299257 5401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:51.311664 5401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05: Generated new TSK 0
I20260927 22:40:51.312958 5401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:40:51.363623 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34021
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:51.699213 5417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:51.699492 5417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:51.699589 5417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:51.706905 5417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:51.707008 5417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:51.707118 5417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:51.717296 5417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:51.718562 5417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:51.719774 5417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:51.732532 5423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:51.732535 5422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:51.732689 5425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:51.733456 5417 server_base.cc:1034] running on GCE node
I20260927 22:40:51.734114 5417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:51.735113 5417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:51.736454 5417 hybrid_clock.cc:648] HybridClock initialized: now 1790548851736407 us; error 34 us; skew 500 ppm
I20260927 22:40:51.736792 5417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:51 dist-test-slave-pp80 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:51.739904 5417 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:40:51.741181 5417 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:51.741274 5417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:51.741613 5417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:51.744594 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "34a1ff0471954aa2a8aa0004f5167001"
format_stamp: "Formatted at 2026-09-27 22:40:51 on dist-test-slave-pp80"
server_key: "4e8eb64f55c5098cc4e2af6148e182d6"
server_key_iv: "dfc4dc0df10da5036bf002ac616b49ce"
server_key_version: "encryptionkey@0"
I20260927 22:40:51.745385 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "34a1ff0471954aa2a8aa0004f5167001"
format_stamp: "Formatted at 2026-09-27 22:40:51 on dist-test-slave-pp80"
server_key: "4e8eb64f55c5098cc4e2af6148e182d6"
server_key_iv: "dfc4dc0df10da5036bf002ac616b49ce"
server_key_version: "encryptionkey@0"
I20260927 22:40:51.750560 5417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:40:51.754038 5431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:51.755411 5417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:40:51.755587 5417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "34a1ff0471954aa2a8aa0004f5167001"
format_stamp: "Formatted at 2026-09-27 22:40:51 on dist-test-slave-pp80"
server_key: "4e8eb64f55c5098cc4e2af6148e182d6"
server_key_iv: "dfc4dc0df10da5036bf002ac616b49ce"
server_key_version: "encryptionkey@0"
I20260927 22:40:51.755831 5417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:51.769310 5417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:51.773644 5417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:51.773932 5417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:51.774878 5417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:51.776206 5417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:51.776393 5417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:51.776525 5417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:51.776609 5417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:51.809461 5417 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:40647
I20260927 22:40:51.809510 5543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:40647 every 8 connection(s)
I20260927 22:40:51.810362 5417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:51.814149 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5417
I20260927 22:40:51.814373 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:51.815079 1237 external_mini_cluster.cc:1428] Setting key 64a49c657fef23a6eec8854b62cba8fc
Sep 27 22:40:51 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:51.829897 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:51.812876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:48729 (local address 127.1.53.126:34021)
0927 22:40:51.813275 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:51.813280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:51.814415 (+ 1135us) server_negotiation.cc:407] Connection header received
0927 22:40:51.816699 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:51.816704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:51.816798 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:51.816926 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:51.818409 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:51.819100 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:51.820697 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:51.821100 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:51.821229 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:51.824537 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:51.824544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:51.824553 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:51.824583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:51.826980 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:51.827678 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:51.827687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:51.827690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:51.827795 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:51.828200 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:51.828204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:51.828207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:51.828519 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:51.828682 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:51.829336 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 22:40:51.829541 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":209,"threads_started":1}
I20260927 22:40:51.830880 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:51.813478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:34021 (local address 127.1.53.65:48729)
0927 22:40:51.814216 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:51.814284 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:51.816435 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:51.817161 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:51.817171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:51.817693 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:51.818242 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:51.818258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:51.819263 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:51.819267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:51.820548 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:51.820559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:51.821320 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:51.821327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:51.821538 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:51.822209 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:51.822235 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:51.824338 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:51.827122 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:51.827130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:51.827147 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:51.827555 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:51.827907 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:51.827912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:51.827916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:51.828086 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:51.828642 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:51.828650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:51.828981 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 22:40:51.829232 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 22:40:51.829736 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:40:51.832515 5544 heartbeater.cc:347] Connected to a master server at 127.1.53.126:34021
I20260927 22:40:51.832854 5544 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:51.833606 5544 heartbeater.cc:511] Master 127.1.53.126:34021 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:40:51.835739 5355 ts_manager.cc:194] Registered new tserver with Master: 34a1ff0471954aa2a8aa0004f5167001 (127.1.53.65:40647)
I20260927 22:40:51.837939 5355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:48729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:40:51.861449 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34021
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:52.191429 5550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:52.191737 5550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:52.191910 5550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:52.199350 5550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:52.199476 5550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:52.199667 5550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:52.209741 5550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:52.211084 5550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:52.212563 5550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:52.232633 5558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:52.233104 5556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:52.233237 5555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:52.234030 5550 server_base.cc:1034] running on GCE node
I20260927 22:40:52.234640 5550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:52.235654 5550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:52.236899 5550 hybrid_clock.cc:648] HybridClock initialized: now 1790548852236844 us; error 38 us; skew 500 ppm
I20260927 22:40:52.237224 5550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:52 dist-test-slave-pp80 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:52.240417 5550 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:40:52.241569 5550 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:52.241657 5550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:52.241900 5550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:52.244678 5550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "c82beceb7cf948e5a8dc5ac673482657"
format_stamp: "Formatted at 2026-09-27 22:40:52 on dist-test-slave-pp80"
server_key: "1a119dc6d214ec3319326e4c54a6a7e3"
server_key_iv: "3f0577d9b858a888e2b0bcbe0cc8d895"
server_key_version: "encryptionkey@0"
I20260927 22:40:52.245396 5550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "c82beceb7cf948e5a8dc5ac673482657"
format_stamp: "Formatted at 2026-09-27 22:40:52 on dist-test-slave-pp80"
server_key: "1a119dc6d214ec3319326e4c54a6a7e3"
server_key_iv: "3f0577d9b858a888e2b0bcbe0cc8d895"
server_key_version: "encryptionkey@0"
I20260927 22:40:52.250900 5550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:40:52.265095 5564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:52.268220 5550 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.007s
I20260927 22:40:52.268487 5550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "c82beceb7cf948e5a8dc5ac673482657"
format_stamp: "Formatted at 2026-09-27 22:40:52 on dist-test-slave-pp80"
server_key: "1a119dc6d214ec3319326e4c54a6a7e3"
server_key_iv: "3f0577d9b858a888e2b0bcbe0cc8d895"
server_key_version: "encryptionkey@0"
I20260927 22:40:52.268743 5550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:52.299255 5550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:52.303467 5550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:52.303745 5550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:52.304803 5550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:52.306151 5550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:52.306237 5550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:52.306320 5550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:52.306383 5550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:52.338500 5550 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:37495
I20260927 22:40:52.338596 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:37495 every 8 connection(s)
I20260927 22:40:52.339398 5550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:52.345790 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5550
I20260927 22:40:52.345999 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:52.346477 1237 external_mini_cluster.cc:1428] Setting key 303bb7ecf83ec619331844667e8c8dc9
Sep 27 22:40:52 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:52.360306 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.341853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:47243 (local address 127.1.53.126:34021)
0927 22:40:52.342283 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:52.342290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:52.343364 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 22:40:52.345191 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:52.345195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:52.345267 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:52.345363 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:52.347118 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.347837 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.349599 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.350397 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.350560 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.354408 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:52.354413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:52.354417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:52.354448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:52.357154 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:52.357880 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:52.357887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:52.357895 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:52.358021 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:52.358425 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:52.358431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:52.358434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:52.358686 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:52.358836 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:52.359648 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 22:40:52.359876 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260927 22:40:52.361445 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.342438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:34021 (local address 127.1.53.66:47243)
0927 22:40:52.343195 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:52.343242 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:52.344937 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:52.345580 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:52.345589 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:52.346293 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:52.346911 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.346949 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.347991 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.347999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.349449 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.349465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.350634 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.350639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.350899 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.351627 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:52.351649 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:52.354169 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:52.357299 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:52.357308 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:52.357326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:52.357750 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:52.358131 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:52.358135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:52.358139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:52.358310 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:52.358819 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:52.358827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:52.359174 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 22:40:52.359555 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 22:40:52.360130 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":349,"threads_started":1}
I20260927 22:40:52.363010 5677 heartbeater.cc:347] Connected to a master server at 127.1.53.126:34021
I20260927 22:40:52.363318 5677 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:52.363957 5677 heartbeater.cc:511] Master 127.1.53.126:34021 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
I20260927 22:40:52.365482 5355 ts_manager.cc:194] Registered new tserver with Master: c82beceb7cf948e5a8dc5ac673482657 (127.1.53.66:37495)
I20260927 22:40:52.366299 5355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:47243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:40:52.391294 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34021
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:40:52.715070 5683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:52.715358 5683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:52.715454 5683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:52.722940 5683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:52.723068 5683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:52.723186 5683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:52.732915 5683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:52.734144 5683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:52.735373 5683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:52.747987 5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:52.747951 5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:52.748024 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:52.749315 5683 server_base.cc:1034] running on GCE node
I20260927 22:40:52.749841 5683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:52.750741 5683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:52.751965 5683 hybrid_clock.cc:648] HybridClock initialized: now 1790548852751912 us; error 50 us; skew 500 ppm
I20260927 22:40:52.752358 5683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:40:52 dist-test-slave-pp80 krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:40:52.755435 5683 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:40:52.756705 5683 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:52.756839 5683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:52.757164 5683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:52.760461 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "57fe91ef890b46c4b561de7db77851b3"
format_stamp: "Formatted at 2026-09-27 22:40:52 on dist-test-slave-pp80"
server_key: "291d07131b60763a98615b7763d4dcd8"
server_key_iv: "7544d73b4d47f1f4f0710f02f0ba4a9d"
server_key_version: "encryptionkey@0"
I20260927 22:40:52.761334 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "57fe91ef890b46c4b561de7db77851b3"
format_stamp: "Formatted at 2026-09-27 22:40:52 on dist-test-slave-pp80"
server_key: "291d07131b60763a98615b7763d4dcd8"
server_key_iv: "7544d73b4d47f1f4f0710f02f0ba4a9d"
server_key_version: "encryptionkey@0"
I20260927 22:40:52.766517 5683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:40:52.770391 5697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:52.771775 5683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:40:52.771947 5683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "57fe91ef890b46c4b561de7db77851b3"
format_stamp: "Formatted at 2026-09-27 22:40:52 on dist-test-slave-pp80"
server_key: "291d07131b60763a98615b7763d4dcd8"
server_key_iv: "7544d73b4d47f1f4f0710f02f0ba4a9d"
server_key_version: "encryptionkey@0"
I20260927 22:40:52.772169 5683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:52.798569 5683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:52.803220 5683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:52.803514 5683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:52.804589 5683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:52.805968 5683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:52.806092 5683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:52.806231 5683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:52.806318 5683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:52.840900 5683 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:34425
I20260927 22:40:52.840981 5809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:34425 every 8 connection(s)
I20260927 22:40:52.841169 5544 heartbeater.cc:503] Master 127.1.53.126:34021 was elected leader, sending a full tablet report...
I20260927 22:40:52.841917 5683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:52.850927 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5683
I20260927 22:40:52.851151 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:52.851691 1237 external_mini_cluster.cc:1428] Setting key 03372d39314a5c10b24b715d49fef6f2
Sep 27 22:40:52 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:40:52.863085 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.844486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:38553 (local address 127.1.53.126:34021)
0927 22:40:52.844650 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:52.844655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:52.845840 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 22:40:52.847659 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:52.847663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:52.847743 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:52.847832 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:52.849461 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.850220 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.852705 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.853180 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.853333 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.857023 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:52.857029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:52.857033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:52.857069 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:52.859912 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:52.860861 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:52.860865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:52.860868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:52.861024 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:52.861399 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:52.861401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:52.861403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:52.861674 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:52.861785 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:52.862489 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 22:40:52.862709 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:40:52.864147 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.844969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:34021 (local address 127.1.53.67:38553)
0927 22:40:52.845664 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:52.845709 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:52.847388 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:52.848100 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:52.848109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:52.848673 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:52.849270 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.849288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.850396 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.850400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.852546 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.852560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.853383 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.853388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.853658 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.854375 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:40:52.854397 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:52.856798 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:52.860057 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:52.860066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:52.860080 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:52.860748 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:52.861125 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:40:52.861128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:40:52.861130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:40:52.861304 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:40:52.861779 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:52.861786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:52.862119 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 22:40:52.862399 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 22:40:52.862942 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":298,"threads_started":1}
I20260927 22:40:52.865707 5810 heartbeater.cc:347] Connected to a master server at 127.1.53.126:34021
I20260927 22:40:52.866073 5810 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:52.866863 5810 heartbeater.cc:511] Master 127.1.53.126:34021 requested a full tablet report, sending...
I20260927 22:40:52.868559 5355 ts_manager.cc:194] Registered new tserver with Master: 57fe91ef890b46c4b561de7db77851b3 (127.1.53.67:34425)
I20260927 22:40:52.869362 5355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:38553
I20260927 22:40:52.879729 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:40:52.896648 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.884273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.1.53.126:34021)
0927 22:40:52.884472 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:52.884481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:52.884968 (+ 487us) server_negotiation.cc:407] Connection header received
0927 22:40:52.885089 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:52.885093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:52.885157 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:52.885242 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:52.886662 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.887408 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.888726 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.889232 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.889362 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.891549 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:52.891554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:52.891558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:52.891595 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:52.894142 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:52.894753 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:52.894757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:52.894760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:52.894910 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:52.895250 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:52.895254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:52.895256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:52.895579 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:52.895763 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:52.896045 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 22:40:52.896225 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:40:52.900777 5355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:52.907502 5355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:52.935305 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.920101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.1.53.65:40647)
0927 22:40:52.927025 (+ 6924us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:52.927034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:52.927066 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:40:52.927166 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:52.927174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:52.927480 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:52.928258 (+ 778us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:52.928949 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.930550 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.933058 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.934180 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.934327 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.934568 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:52.934786 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 22:40:52.934950 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6718,"thread_start_us":6522,"threads_started":1}
I20260927 22:40:52.935351 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.919653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:40647 (local address 127.0.0.1:57018)
0927 22:40:52.925829 (+ 6176us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:52.925849 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:52.925963 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:52.927736 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:52.927742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:52.927777 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:52.928722 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.928737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.930690 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.930695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.932871 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.932884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.934356 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.934362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.934545 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.934597 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 22:40:52.934732 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:40:52.934970 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5995,"mutex_wait_us":90,"thread_start_us":5857,"threads_started":1}
I20260927 22:40:52.935890 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.919424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:34425 (local address 127.0.0.1:56704)
0927 22:40:52.920274 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:52.920327 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:52.920528 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:52.927374 (+ 6846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:52.927381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:52.927414 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:52.927942 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.927961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.930275 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.930281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.932628 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.932643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.934962 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.934969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.935156 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.935190 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:40:52.935282 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:40:52.935448 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":3,"thread_start_us":948,"threads_started":1}
I20260927 22:40:52.936738 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.919732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.1.53.67:34425)
0927 22:40:52.926589 (+ 6857us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:52.926602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:52.926645 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:40:52.926763 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:52.926775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:52.927085 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:52.927321 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:52.928214 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.930099 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.932849 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.934473 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.934592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.934811 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:52.936179 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0927 22:40:52.936393 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6645,"thread_start_us":6473,"threads_started":1}
I20260927 22:40:52.937338 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.927255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:37495 (local address 127.0.0.1:51116)
0927 22:40:52.928650 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:52.929079 (+ 429us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:52.929187 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:52.929695 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:52.929700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:52.929724 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:52.930064 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.930072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.931695 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.931704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.934191 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:52.934206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.936454 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.936461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:52.936647 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.936681 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:40:52.936792 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:40:52.936945 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":67,"thread_start_us":553,"threads_started":1}
I20260927 22:40:52.937803 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:52.926149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.1.53.66:37495)
0927 22:40:52.927193 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:52.927201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:52.929303 (+ 2102us) server_negotiation.cc:407] Connection header received
0927 22:40:52.929392 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:52.929397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:52.929536 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:52.929714 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:52.930202 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.931538 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.934390 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:52.935956 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:52.936101 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:52.936323 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:52.937271 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 22:40:52.937489 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":791,"threads_started":1}
I20260927 22:40:52.939798 5479 tablet_service.cc:1467] Processing CreateTablet for tablet aae3e4f7ce554baabeaf53302b066929 (DEFAULT_TABLE table=test-table [id=50a832ecffc245a9aa8b2704c2373ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:52.940342 5745 tablet_service.cc:1467] Processing CreateTablet for tablet aae3e4f7ce554baabeaf53302b066929 (DEFAULT_TABLE table=test-table [id=50a832ecffc245a9aa8b2704c2373ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:52.940830 5612 tablet_service.cc:1467] Processing CreateTablet for tablet aae3e4f7ce554baabeaf53302b066929 (DEFAULT_TABLE table=test-table [id=50a832ecffc245a9aa8b2704c2373ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:52.941329 5479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae3e4f7ce554baabeaf53302b066929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:52.941659 5745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae3e4f7ce554baabeaf53302b066929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:52.942173 5612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae3e4f7ce554baabeaf53302b066929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:52.953426 5828 tablet_bootstrap.cc:492] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Bootstrap starting.
I20260927 22:40:52.955458 5829 tablet_bootstrap.cc:492] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Bootstrap starting.
I20260927 22:40:52.956195 5830 tablet_bootstrap.cc:492] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Bootstrap starting.
I20260927 22:40:52.957235 5828 tablet_bootstrap.cc:654] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:52.958567 5828 log.cc:824] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:52.959766 5830 tablet_bootstrap.cc:654] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:52.961028 5830 log.cc:824] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:52.961917 5829 tablet_bootstrap.cc:654] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:52.962535 5828 tablet_bootstrap.cc:492] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: No bootstrap required, opened a new log
I20260927 22:40:52.962805 5828 ts_tablet_manager.cc:1397] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:40:52.962994 5829 log.cc:824] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:52.964000 5830 tablet_bootstrap.cc:492] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: No bootstrap required, opened a new log
I20260927 22:40:52.964285 5830 ts_tablet_manager.cc:1397] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:40:52.965466 5829 tablet_bootstrap.cc:492] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: No bootstrap required, opened a new log
I20260927 22:40:52.965718 5829 ts_tablet_manager.cc:1397] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 22:40:52.973510 5829 raft_consensus.cc:348] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:52.973600 5828 raft_consensus.cc:348] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:52.973937 5829 raft_consensus.cc:374] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:52.974064 5829 raft_consensus.cc:729] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fe91ef890b46c4b561de7db77851b3, State: Initialized, Role: FOLLOWER
I20260927 22:40:52.974052 5828 raft_consensus.cc:374] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:52.974167 5828 raft_consensus.cc:729] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82beceb7cf948e5a8dc5ac673482657, State: Initialized, Role: FOLLOWER
I20260927 22:40:52.974687 5829 consensus_queue.cc:260] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:52.974843 5828 consensus_queue.cc:260] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:52.974738 5830 raft_consensus.cc:348] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:52.975139 5830 raft_consensus.cc:374] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:52.975241 5830 raft_consensus.cc:729] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a1ff0471954aa2a8aa0004f5167001, State: Initialized, Role: FOLLOWER
I20260927 22:40:52.976122 5830 consensus_queue.cc:260] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:52.977380 5677 heartbeater.cc:503] Master 127.1.53.126:34021 was elected leader, sending a full tablet report...
I20260927 22:40:52.977485 5810 heartbeater.cc:503] Master 127.1.53.126:34021 was elected leader, sending a full tablet report...
I20260927 22:40:52.977708 5829 ts_tablet_manager.cc:1428] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260927 22:40:52.979902 5828 ts_tablet_manager.cc:1428] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260927 22:40:52.981455 5830 ts_tablet_manager.cc:1428] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260927 22:40:53.014323 5836 raft_consensus.cc:482] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:53.014665 5836 raft_consensus.cc:504] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:53.016439 5836 leader_election.cc:288] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57fe91ef890b46c4b561de7db77851b3 (127.1.53.67:34425), c82beceb7cf948e5a8dc5ac673482657 (127.1.53.66:37495)
I20260927 22:40:53.023644 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.016964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:36821 (local address 127.1.53.66:37495)
0927 22:40:53.017153 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:53.017161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:53.017938 (+ 777us) server_negotiation.cc:407] Connection header received
0927 22:40:53.018111 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:53.018114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:53.018168 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:53.018273 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:53.019019 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.019892 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.021819 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.022609 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.022711 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.022776 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:53.023210 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 22:40:53.023362 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:40:53.023644 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.016867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:49393 (local address 127.1.53.67:34425)
0927 22:40:53.017067 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:53.017075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:53.017851 (+ 776us) server_negotiation.cc:407] Connection header received
0927 22:40:53.018050 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:53.018059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:53.018138 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:53.018205 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:40:53.018901 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.019860 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.021777 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.022780 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.022910 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.023009 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:53.023265 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:40:53.023403 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:40:53.023753 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.016949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:37495 (local address 127.1.53.65:36821)
0927 22:40:53.017787 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:53.017824 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:53.017981 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:53.018305 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:53.018310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:53.018339 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:53.018837 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:53.018852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.020062 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.020066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:53.021636 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:53.021647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.022820 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.022828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:53.023012 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.023052 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:40:53.023159 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:40:53.023297 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":33,"thread_start_us":531,"threads_started":1}
I20260927 22:40:53.023779 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.016896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:34425 (local address 127.1.53.65:49393)
0927 22:40:53.017730 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:53.017745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:53.017872 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:53.018258 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:53.018263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:53.018285 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:40:53.018751 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:53.018764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.019997 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.020002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:53.021629 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:53.021640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.022976 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.022980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:53.023099 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.023120 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:40:53.023229 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:40:53.023361 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":6,"thread_start_us":464,"threads_started":1}
I20260927 22:40:53.024866 5765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae3e4f7ce554baabeaf53302b066929" candidate_uuid: "34a1ff0471954aa2a8aa0004f5167001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe91ef890b46c4b561de7db77851b3" is_pre_election: true
I20260927 22:40:53.025053 5632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae3e4f7ce554baabeaf53302b066929" candidate_uuid: "34a1ff0471954aa2a8aa0004f5167001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82beceb7cf948e5a8dc5ac673482657" is_pre_election: true
I20260927 22:40:53.025403 5765 raft_consensus.cc:2454] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a1ff0471954aa2a8aa0004f5167001 in term 0.
I20260927 22:40:53.025583 5632 raft_consensus.cc:2454] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34a1ff0471954aa2a8aa0004f5167001 in term 0.
I20260927 22:40:53.026067 5432 leader_election.cc:302] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a1ff0471954aa2a8aa0004f5167001, 57fe91ef890b46c4b561de7db77851b3; no voters:
I20260927 22:40:53.026466 5836 raft_consensus.cc:2781] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:53.026624 5836 raft_consensus.cc:482] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:53.026726 5836 raft_consensus.cc:3037] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:53.028940 5836 raft_consensus.cc:504] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:53.029547 5836 leader_election.cc:288] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [CANDIDATE]: Term 1 election: Requested vote from peers 57fe91ef890b46c4b561de7db77851b3 (127.1.53.67:34425), c82beceb7cf948e5a8dc5ac673482657 (127.1.53.66:37495)
I20260927 22:40:53.029910 5765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae3e4f7ce554baabeaf53302b066929" candidate_uuid: "34a1ff0471954aa2a8aa0004f5167001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fe91ef890b46c4b561de7db77851b3"
I20260927 22:40:53.029979 5632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae3e4f7ce554baabeaf53302b066929" candidate_uuid: "34a1ff0471954aa2a8aa0004f5167001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82beceb7cf948e5a8dc5ac673482657"
I20260927 22:40:53.030119 5765 raft_consensus.cc:3037] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:53.030149 5632 raft_consensus.cc:3037] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:53.033262 5632 raft_consensus.cc:2454] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a1ff0471954aa2a8aa0004f5167001 in term 1.
I20260927 22:40:53.033262 5765 raft_consensus.cc:2454] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34a1ff0471954aa2a8aa0004f5167001 in term 1.
I20260927 22:40:53.033659 5435 leader_election.cc:302] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a1ff0471954aa2a8aa0004f5167001, c82beceb7cf948e5a8dc5ac673482657; no voters:
I20260927 22:40:53.033955 5836 raft_consensus.cc:2781] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:53.034349 5836 raft_consensus.cc:686] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 1 LEADER]: Becoming Leader. State: Replica: 34a1ff0471954aa2a8aa0004f5167001, State: Running, Role: LEADER
I20260927 22:40:53.034817 5836 consensus_queue.cc:237] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } }
I20260927 22:40:53.039597 5352 catalog_manager.cc:5495] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34a1ff0471954aa2a8aa0004f5167001 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "34a1ff0471954aa2a8aa0004f5167001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a1ff0471954aa2a8aa0004f5167001" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 } health_report { overall_health: UNKNOWN } } }
I20260927 22:40:53.055873 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.050680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.1.53.65:40647)
0927 22:40:53.050879 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:53.050887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:53.051044 (+ 157us) server_negotiation.cc:407] Connection header received
0927 22:40:53.051239 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:53.051243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:53.051347 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:53.051486 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:40:53.051807 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.052612 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.053846 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.054249 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.054357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.054800 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:40:53.055333 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:40:53.055468 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:53.055505 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 22:40:53.055566 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:40:53.062616 5545 tablet.cc:2369] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:40:53.064209 5632 raft_consensus.cc:1261] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 1 FOLLOWER]: Refusing update from remote peer 34a1ff0471954aa2a8aa0004f5167001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:53.064208 5765 raft_consensus.cc:1261] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 1 FOLLOWER]: Refusing update from remote peer 34a1ff0471954aa2a8aa0004f5167001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:53.065340 5839 consensus_queue.cc:1035] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fe91ef890b46c4b561de7db77851b3" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:53.065743 5836 consensus_queue.cc:1035] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82beceb7cf948e5a8dc5ac673482657" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:53.077474 5843 mvcc.cc:205] Tried to move back new op lower bound from 7334088102140481536 to 7334088102034231296. Current Snapshot: MvccSnapshot[applied={T|T < 7334088102140481536}]
I20260927 22:40:53.082206 5844 mvcc.cc:205] Tried to move back new op lower bound from 7334088102140481536 to 7334088102034231296. Current Snapshot: MvccSnapshot[applied={T|T < 7334088102140481536 or (T in {7334088102140481536})}]
I20260927 22:40:53.094334 5352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:53.094609 5352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:53.097918 5352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee8a0cec76fa467f807cda3c16902d05: Sending DeleteTablet for 3 replicas of tablet aae3e4f7ce554baabeaf53302b066929
I20260927 22:40:53.099035 5745 tablet_service.cc:1514] Processing DeleteTablet for tablet aae3e4f7ce554baabeaf53302b066929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:53 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:56704
I20260927 22:40:53.099346 5479 tablet_service.cc:1514] Processing DeleteTablet for tablet aae3e4f7ce554baabeaf53302b066929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:53 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:57018
I20260927 22:40:53.100020 5852 tablet_replica.cc:330] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: stopping tablet replica
I20260927 22:40:53.100069 5612 tablet_service.cc:1514] Processing DeleteTablet for tablet aae3e4f7ce554baabeaf53302b066929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:53 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:51116
I20260927 22:40:53.100260 5853 tablet_replica.cc:330] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: stopping tablet replica
I20260927 22:40:53.100446 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5417
I20260927 22:40:53.100637 5853 raft_consensus.cc:2229] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:53.100687 5852 raft_consensus.cc:2229] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:53.101303 5852 raft_consensus.cc:2258] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:53.101321 5853 raft_consensus.cc:2258] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:53.103564 5853 ts_tablet_manager.cc:1905] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:53.103564 5852 ts_tablet_manager.cc:1905] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:53.111796 5854 tablet_replica.cc:330] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: stopping tablet replica
I20260927 22:40:53.111826 5852 ts_tablet_manager.cc:1918] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:53.112140 5852 log.cc:1196] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/aae3e4f7ce554baabeaf53302b066929
I20260927 22:40:53.112257 5854 raft_consensus.cc:2229] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:53.112703 5853 ts_tablet_manager.cc:1918] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:53.112792 5854 raft_consensus.cc:2258] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:53.112859 5853 log.cc:1196] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/aae3e4f7ce554baabeaf53302b066929
I20260927 22:40:53.113152 5852 ts_tablet_manager.cc:1939] T aae3e4f7ce554baabeaf53302b066929 P 57fe91ef890b46c4b561de7db77851b3: Deleting consensus metadata
I20260927 22:40:53.113871 5853 ts_tablet_manager.cc:1939] T aae3e4f7ce554baabeaf53302b066929 P 34a1ff0471954aa2a8aa0004f5167001: Deleting consensus metadata
I20260927 22:40:53.115154 5339 catalog_manager.cc:4841] TS 57fe91ef890b46c4b561de7db77851b3 (127.1.53.67:34425): tablet aae3e4f7ce554baabeaf53302b066929 (table test-table [id=50a832ecffc245a9aa8b2704c2373ea4]) successfully deleted
I20260927 22:40:53.115496 5342 catalog_manager.cc:4841] TS 34a1ff0471954aa2a8aa0004f5167001 (127.1.53.65:40647): tablet aae3e4f7ce554baabeaf53302b066929 (table test-table [id=50a832ecffc245a9aa8b2704c2373ea4]) successfully deleted
I20260927 22:40:53.116504 5854 ts_tablet_manager.cc:1905] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:53.122792 5854 ts_tablet_manager.cc:1918] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:53.122985 5854 log.cc:1196] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/aae3e4f7ce554baabeaf53302b066929
I20260927 22:40:53.123564 5854 ts_tablet_manager.cc:1939] T aae3e4f7ce554baabeaf53302b066929 P c82beceb7cf948e5a8dc5ac673482657: Deleting consensus metadata
Sep 27 22:40:53 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:40:53.125363 5342 catalog_manager.cc:4841] TS c82beceb7cf948e5a8dc5ac673482657 (127.1.53.66:37495): tablet aae3e4f7ce554baabeaf53302b066929 (table test-table [id=50a832ecffc245a9aa8b2704c2373ea4]) successfully deleted
I20260927 22:40:53.132630 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.102611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.1.53.65:40647)
0927 22:40:53.102777 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:53.102785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:53.113896 (+ 11111us) server_negotiation.cc:407] Connection header received
0927 22:40:53.114162 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:53.114168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:53.114250 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:53.114395 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:53.116782 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.118021 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.119894 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.120660 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.120788 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.126015 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:53.126026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:53.126040 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:53.126080 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:53.129502 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:53.130424 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:53.130431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:53.130433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:53.130593 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:53.131003 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:53.131006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:53.131009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:53.131356 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:53.131591 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:53.132007 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:40:53.132189 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:40:53.133374 5539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:53.231201 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5417
I20260927 22:40:53.249009 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5550
Sep 27 22:40:53 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:40:53.263576 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.249794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.1.53.66:37495)
0927 22:40:53.249969 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:53.249973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:53.249991 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:40:53.250110 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:53.250114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:53.250204 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:53.250296 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:53.251457 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.252187 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.253460 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.253982 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.254089 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.258291 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:53.258304 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:53.258321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:53.258362 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:53.261205 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:53.261808 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:53.261815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:53.261817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:53.261944 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:53.262243 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:53.262246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:53.262248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:53.262610 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:53.262827 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:53.263033 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:40:53.263194 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:40:53.264495 5672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:53.353310 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5550
I20260927 22:40:53.370287 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5683
Sep 27 22:40:53 dist-test-slave-pp80 krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:40:53.385609 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:53.370993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.1.53.67:34425)
0927 22:40:53.371185 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:53.371190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:53.371208 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:40:53.371340 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:53.371344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:53.371417 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:53.371513 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:53.372843 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.373578 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.374980 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:53.375379 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:53.375468 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:53.380048 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:53.380064 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:53.380080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:40:53.380116 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:53.382972 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:53.383607 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:53.383615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:53.383617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:53.383751 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:40:53.384122 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:40:53.384126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:40:53.384128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:40:53.384604 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:53.384805 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:53.385112 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:40:53.385268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:40:53.386274 5805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:53.474607 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5683
I20260927 22:40:53.493458 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5324
I20260927 22:40:53.494736 5385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:53.602252 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5324
2026-09-27T22:40:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4110 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:53Z Disabled control of system clock
I20260927 22:40:53.669556 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:34641
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:40:53.996330 5863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:53.996627 5863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:53.996796 5863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:54.004061 5863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:54.004179 5863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:54.004276 5863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:54.004369 5863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:54.004438 5863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:40:54.013765 5863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:34641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:54.015223 5863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:54.017143 5863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:54.032716 5871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:54.032729 5868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:54.032657 5869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:54.033800 5863 server_base.cc:1034] running on GCE node
I20260927 22:40:54.034638 5863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:54.036074 5863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:54.037307 5863 hybrid_clock.cc:648] HybridClock initialized: now 1790548854037244 us; error 63 us; skew 500 ppm
I20260927 22:40:54.037858 5863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:54.046142 5863 webserver.cc:460] Webserver started at http://127.1.53.126:37953/ using document root <none> and password file <none>
I20260927 22:40:54.046837 5863 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:54.046931 5863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:54.047274 5863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:54.050246 5863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "09c01c34d5e148f3a2ac8af20ab05c7d"
format_stamp: "Formatted at 2026-09-27 22:40:54 on dist-test-slave-pp80"
server_key: "3461e155103c2870c447b50ea3be6261"
server_key_iv: "4911e19124b3361a03dcdf030f7a89b2"
server_key_version: "encryptionkey@0"
I20260927 22:40:54.050997 5863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "09c01c34d5e148f3a2ac8af20ab05c7d"
format_stamp: "Formatted at 2026-09-27 22:40:54 on dist-test-slave-pp80"
server_key: "3461e155103c2870c447b50ea3be6261"
server_key_iv: "4911e19124b3361a03dcdf030f7a89b2"
server_key_version: "encryptionkey@0"
I20260927 22:40:54.056391 5863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 22:40:54.060122 5877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:54.061753 5863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:54.061962 5863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "09c01c34d5e148f3a2ac8af20ab05c7d"
format_stamp: "Formatted at 2026-09-27 22:40:54 on dist-test-slave-pp80"
server_key: "3461e155103c2870c447b50ea3be6261"
server_key_iv: "4911e19124b3361a03dcdf030f7a89b2"
server_key_version: "encryptionkey@0"
I20260927 22:40:54.062168 5863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:54.068333 5863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:54.069103 5863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:54.069356 5863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:54.087584 5863 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:34641
I20260927 22:40:54.087620 5928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:34641 every 8 connection(s)
I20260927 22:40:54.089013 5863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:54.092767 5929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:54.098416 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5863
I20260927 22:40:54.098618 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:54.099427 1237 external_mini_cluster.cc:1428] Setting key 1e4bcb7f3a16025aee6d9f248994484b
I20260927 22:40:54.104338 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d: Bootstrap starting.
I20260927 22:40:54.107764 5929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:54.108621 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:54.101847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.1.53.126:34641)
0927 22:40:54.102855 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:54.102866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:54.102952 (+ 86us) server_negotiation.cc:407] Connection header received
0927 22:40:54.104735 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:54.104759 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:54.105240 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:54.105642 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:54.106775 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:54.106781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:54.106794 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:54.106806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:54.106862 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:54.106887 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:54.107084 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:54.107224 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 22:40:54.107486 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":333,"threads_started":1}
I20260927 22:40:54.108979 5929 log.cc:824] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:54.111898 5929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d: No bootstrap required, opened a new log
I20260927 22:40:54.119781 5929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34641 } }
I20260927 22:40:54.120188 5929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:54.120319 5929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c01c34d5e148f3a2ac8af20ab05c7d, State: Initialized, Role: FOLLOWER
I20260927 22:40:54.120884 5929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34641 } }
I20260927 22:40:54.121134 5929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:54.121294 5929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:54.121486 5929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:54.123644 5929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34641 } }
I20260927 22:40:54.124151 5929 leader_election.cc:302] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c01c34d5e148f3a2ac8af20ab05c7d; no voters:
I20260927 22:40:54.124804 5929 leader_election.cc:288] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:54.125023 5934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:54.125842 5934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [term 1 LEADER]: Becoming Leader. State: Replica: 09c01c34d5e148f3a2ac8af20ab05c7d, State: Running, Role: LEADER
I20260927 22:40:54.126502 5934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34641 } }
I20260927 22:40:54.127130 5929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:54.129298 5935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34641 } } }
I20260927 22:40:54.129637 5935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:54.129951 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c01c34d5e148f3a2ac8af20ab05c7d. Latest consensus state: current_term: 1 leader_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c01c34d5e148f3a2ac8af20ab05c7d" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34641 } } }
I20260927 22:40:54.130515 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:54.135488 5940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:54.140873 5940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:54.150331 5940 catalog_manager.cc:1347] Generated new cluster ID: e33e48b283f44cab82666ef7c1bc0485
I20260927 22:40:54.150573 5940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:54.165549 5940 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:54.166122 5940 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:54.188516 5940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d: Generated new TSK 0
I20260927 22:40:54.189752 5940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:40:54.208602 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34641
--builtin_ntp_servers=127.1.53.84:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:40:54.545791 5953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:54.546131 5953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:54.546290 5953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:54.553514 5953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:54.553666 5953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:54.553858 5953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:40:54.563241 5953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:54.564702 5953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:54.566833 5953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:54.580677 5959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:54.580601 5958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:54.581071 5961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:54.581624 5953 server_base.cc:1034] running on GCE node
I20260927 22:40:54.582165 5953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:54.583076 5953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:54.584307 5953 hybrid_clock.cc:648] HybridClock initialized: now 1790548854584202 us; error 54 us; skew 500 ppm
I20260927 22:40:54.584623 5953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:54.586858 5953 webserver.cc:460] Webserver started at http://127.1.53.65:39847/ using document root <none> and password file <none>
I20260927 22:40:54.587536 5953 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:54.587648 5953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:54.587960 5953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:54.590832 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "4a9d6eb070374571ad4f6890c3256fed"
format_stamp: "Formatted at 2026-09-27 22:40:54 on dist-test-slave-pp80"
server_key: "ca12895d77dfb3e950ffabfc6f263736"
server_key_iv: "d66599f4c2f19727dce242fc46e07819"
server_key_version: "encryptionkey@0"
I20260927 22:40:54.591619 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "4a9d6eb070374571ad4f6890c3256fed"
format_stamp: "Formatted at 2026-09-27 22:40:54 on dist-test-slave-pp80"
server_key: "ca12895d77dfb3e950ffabfc6f263736"
server_key_iv: "d66599f4c2f19727dce242fc46e07819"
server_key_version: "encryptionkey@0"
I20260927 22:40:54.596616 5953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:40:54.599866 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:54.601263 5953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:54.601428 5953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "4a9d6eb070374571ad4f6890c3256fed"
format_stamp: "Formatted at 2026-09-27 22:40:54 on dist-test-slave-pp80"
server_key: "ca12895d77dfb3e950ffabfc6f263736"
server_key_iv: "d66599f4c2f19727dce242fc46e07819"
server_key_version: "encryptionkey@0"
I20260927 22:40:54.601652 5953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:54.607540 5953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:54.608603 5953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:54.608839 5953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:54.609725 5953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:54.611074 5953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:54.611160 5953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:54.611233 5953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:54.611289 5953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:54.643591 5953 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:45665
I20260927 22:40:54.643678 6079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:45665 every 8 connection(s)
I20260927 22:40:54.644994 5953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:40:54.650408 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 5953
I20260927 22:40:54.650614 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:40:54.651233 1237 external_mini_cluster.cc:1428] Setting key e038a3775df599c37ad581d6450c1d1c
I20260927 22:40:54.653664 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:54.647752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:50253 (local address 127.1.53.126:34641)
0927 22:40:54.648196 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:54.648201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:54.649477 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 22:40:54.651226 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:54.651231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:54.651305 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:54.651432 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:54.652851 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:54.652856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:54.652860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:54.652874 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:54.652939 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:54.652958 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:54.653106 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:54.653313 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:40:54.653370 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260927 22:40:54.654268 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34641
--builtin_ntp_servers=127.1.53.84:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:40:54.654611 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:54.648429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:34641 (local address 127.1.53.65:50253)
0927 22:40:54.649323 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:54.649358 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:54.650958 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:54.651674 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:54.651682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:54.651773 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:54.652521 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:54.652524 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:54.652572 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:54.652676 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:54.653066 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:54.653076 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:54.653086 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:40:54.653380 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 22:40:54.653584 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":252,"threads_started":1}
I20260927 22:40:54.656114 6080 heartbeater.cc:347] Connected to a master server at 127.1.53.126:34641
I20260927 22:40:54.656459 6080 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:54.657202 6080 heartbeater.cc:511] Master 127.1.53.126:34641 requested a full tablet report, sending...
I20260927 22:40:54.659561 5894 ts_manager.cc:194] Registered new tserver with Master: 4a9d6eb070374571ad4f6890c3256fed (127.1.53.65:45665)
W20260927 22:40:54.995025 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:54.995316 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:54.995466 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:55.002840 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:55.002948 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:55.003096 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:40:55.012547 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:55.014029 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:55.015910 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:55.029539 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:55.029562 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:55.029547 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:55.030474 6084 server_base.cc:1034] running on GCE node
I20260927 22:40:55.031265 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:55.032311 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:55.033574 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790548855033514 us; error 55 us; skew 500 ppm
I20260927 22:40:55.033952 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:55.036535 6084 webserver.cc:460] Webserver started at http://127.1.53.66:44543/ using document root <none> and password file <none>
I20260927 22:40:55.037304 6084 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:55.037400 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:55.037658 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:55.040498 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "98a37e8b740f4e1d8c804377c4400d12"
format_stamp: "Formatted at 2026-09-27 22:40:55 on dist-test-slave-pp80"
server_key: "c36a3430febf891b2abc5292fc3170f3"
server_key_iv: "b0f8413b98f0afcb83f5ef3fe6d106a3"
server_key_version: "encryptionkey@0"
I20260927 22:40:55.041224 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "98a37e8b740f4e1d8c804377c4400d12"
format_stamp: "Formatted at 2026-09-27 22:40:55 on dist-test-slave-pp80"
server_key: "c36a3430febf891b2abc5292fc3170f3"
server_key_iv: "b0f8413b98f0afcb83f5ef3fe6d106a3"
server_key_version: "encryptionkey@0"
I20260927 22:40:55.046355 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:40:55.050333 6098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:55.051937 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:40:55.052151 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "98a37e8b740f4e1d8c804377c4400d12"
format_stamp: "Formatted at 2026-09-27 22:40:55 on dist-test-slave-pp80"
server_key: "c36a3430febf891b2abc5292fc3170f3"
server_key_iv: "b0f8413b98f0afcb83f5ef3fe6d106a3"
server_key_version: "encryptionkey@0"
I20260927 22:40:55.052500 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:55.058295 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:55.059152 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:55.059440 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:55.060527 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:55.062034 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:55.062120 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:55.062258 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:55.062357 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:55.098150 6084 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:37417
I20260927 22:40:55.098184 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:37417 every 8 connection(s)
I20260927 22:40:55.099506 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:40:55.105489 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6084
I20260927 22:40:55.105660 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:40:55.106220 1237 external_mini_cluster.cc:1428] Setting key e9401e1ad495a331009678b8d61b5ad9
I20260927 22:40:55.108785 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.102287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:45449 (local address 127.1.53.126:34641)
0927 22:40:55.102544 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.102549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.104050 (+ 1501us) server_negotiation.cc:407] Connection header received
0927 22:40:55.106362 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.106368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.106445 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.106598 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.107993 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.107998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.108004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.108024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.108065 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.108081 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.108203 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.108460 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.108521 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:40:55.109153 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34641
--builtin_ntp_servers=127.1.53.84:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:40:55.109776 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.102914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:34641 (local address 127.1.53.66:45449)
0927 22:40:55.103870 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.103912 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.105971 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.106871 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.106879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.106984 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.107684 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.107686 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.107731 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.107818 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.108188 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.108196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.108203 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.108489 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.108703 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
I20260927 22:40:55.111246 6211 heartbeater.cc:347] Connected to a master server at 127.1.53.126:34641
I20260927 22:40:55.111585 6211 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:55.112310 6211 heartbeater.cc:511] Master 127.1.53.126:34641 requested a full tablet report, sending...
I20260927 22:40:55.113874 5894 ts_manager.cc:194] Registered new tserver with Master: 98a37e8b740f4e1d8c804377c4400d12 (127.1.53.66:37417)
W20260927 22:40:55.433224 6214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:55.433508 6214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:55.433665 6214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:55.441733 6214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:55.441846 6214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:55.442018 6214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:40:55.451370 6214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:55.452731 6214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:55.454522 6214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:55.468043 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:55.468073 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:55.468021 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:55.469523 6214 server_base.cc:1034] running on GCE node
I20260927 22:40:55.470253 6214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:55.471242 6214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:55.472496 6214 hybrid_clock.cc:648] HybridClock initialized: now 1790548855472447 us; error 60 us; skew 500 ppm
I20260927 22:40:55.472869 6214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:55.475270 6214 webserver.cc:460] Webserver started at http://127.1.53.67:40013/ using document root <none> and password file <none>
I20260927 22:40:55.475977 6214 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:55.476109 6214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:55.476486 6214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:55.479380 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "91c13ea283e14ca09b6d25694c50d2c0"
format_stamp: "Formatted at 2026-09-27 22:40:55 on dist-test-slave-pp80"
server_key: "681011e12cfd3b851989d9e685e48eaa"
server_key_iv: "b6d370046942ee6f6f2b96cde561094d"
server_key_version: "encryptionkey@0"
I20260927 22:40:55.480167 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "91c13ea283e14ca09b6d25694c50d2c0"
format_stamp: "Formatted at 2026-09-27 22:40:55 on dist-test-slave-pp80"
server_key: "681011e12cfd3b851989d9e685e48eaa"
server_key_iv: "b6d370046942ee6f6f2b96cde561094d"
server_key_version: "encryptionkey@0"
I20260927 22:40:55.485567 6214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:40:55.489212 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:55.490597 6214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:40:55.490875 6214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "91c13ea283e14ca09b6d25694c50d2c0"
format_stamp: "Formatted at 2026-09-27 22:40:55 on dist-test-slave-pp80"
server_key: "681011e12cfd3b851989d9e685e48eaa"
server_key_iv: "b6d370046942ee6f6f2b96cde561094d"
server_key_version: "encryptionkey@0"
I20260927 22:40:55.491123 6214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:55.497315 6214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:55.498380 6214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:55.498664 6214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:55.499802 6214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:40:55.501214 6214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:40:55.501300 6214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:55.501379 6214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:40:55.501434 6214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:55.534325 6214 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39377
I20260927 22:40:55.534428 6340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39377 every 8 connection(s)
I20260927 22:40:55.535701 6214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:40:55.539839 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6214
I20260927 22:40:55.540057 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:40:55.540620 1237 external_mini_cluster.cc:1428] Setting key 423a3bcb06d711af33a3f3ccafcea480
I20260927 22:40:55.544880 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.538388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:39923 (local address 127.1.53.126:34641)
0927 22:40:55.538566 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.538570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.540208 (+ 1638us) server_negotiation.cc:407] Connection header received
0927 22:40:55.542558 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.542562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.542652 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.542764 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.544105 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.544109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.544113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.544128 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.544159 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.544171 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.544384 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.544611 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.544685 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:40:55.545833 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.538938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:34641 (local address 127.1.53.67:39923)
0927 22:40:55.540039 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.540092 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.542222 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.542964 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.542973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.543066 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.543803 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.543805 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.543842 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.543924 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.544340 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.544353 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.544365 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.544621 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.544832 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":267,"threads_started":1}
I20260927 22:40:55.547124 6341 heartbeater.cc:347] Connected to a master server at 127.1.53.126:34641
I20260927 22:40:55.547439 6341 heartbeater.cc:464] Registering TS with master...
I20260927 22:40:55.548002 6341 heartbeater.cc:511] Master 127.1.53.126:34641 requested a full tablet report, sending...
I20260927 22:40:55.549474 5894 ts_manager.cc:194] Registered new tserver with Master: 91c13ea283e14ca09b6d25694c50d2c0 (127.1.53.67:39377)
I20260927 22:40:55.555728 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:40:55.562539 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.560104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.1.53.126:34641)
0927 22:40:55.560371 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.560381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.560864 (+ 483us) server_negotiation.cc:407] Connection header received
0927 22:40:55.561017 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.561021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.561085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.561163 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.561976 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.561980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.561983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.561998 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.562028 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.562049 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.562144 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.562184 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.562227 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:40:55.567164 5894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:40:55.574016 5894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:40:55.592191 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.587859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.1.53.65:45665)
0927 22:40:55.589075 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.589088 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.589754 (+ 666us) server_negotiation.cc:407] Connection header received
0927 22:40:55.589824 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.589831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.590078 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.590215 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.591253 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.591265 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.591283 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.591313 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.591370 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.591393 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.591520 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.591723 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.591828 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":890,"threads_started":1}
I20260927 22:40:55.592285 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.586110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.1.53.67:39377)
0927 22:40:55.588765 (+ 2655us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.588772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.588800 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:40:55.588856 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.588862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.589004 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.589852 (+ 848us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.591387 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.591392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.591401 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.591413 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.591489 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.591509 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.591686 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.591791 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.591882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":2324,"threads_started":1}
I20260927 22:40:55.592710 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.585899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39377 (local address 127.0.0.1:46906)
0927 22:40:55.586992 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.587056 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.587211 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.589222 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.589231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.589332 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.590204 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.590207 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.590236 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.590319 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.591622 (+ 1303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.591647 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.591655 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.591813 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.591863 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":566,"threads_started":1}
I20260927 22:40:55.593645 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.586289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:37417 (local address 127.0.0.1:58994)
0927 22:40:55.588404 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.588430 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.588534 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.590570 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.590575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.590598 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.592162 (+ 1564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.592165 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.592182 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.592222 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.593044 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.593048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.593051 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.593213 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.593260 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"thread_start_us":1817,"threads_started":1}
I20260927 22:40:55.593645 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.586733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.1.53.66:37417)
0927 22:40:55.587929 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.587942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.588656 (+ 714us) server_negotiation.cc:407] Connection header received
0927 22:40:55.588735 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.588744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.588999 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.589157 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.592605 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.592616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.592628 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.592658 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.592718 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.592740 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.592979 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.593199 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.593288 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":840,"threads_started":1}
I20260927 22:40:55.594270 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.587331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43794)
0927 22:40:55.589499 (+ 2168us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.589521 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.589629 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.590279 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.590286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.590308 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.591000 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.591005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.591028 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.591073 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.591517 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.591529 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.591538 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.591719 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.591818 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2041,"thread_start_us":1910,"threads_started":1}
I20260927 22:40:55.596940 6276 tablet_service.cc:1467] Processing CreateTablet for tablet 573a07d8a0524ecd9d397f18ae757850 (DEFAULT_TABLE table=test-table [id=834f7a112846405e9fe07a09cf9b4b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:55.597604 6146 tablet_service.cc:1467] Processing CreateTablet for tablet 573a07d8a0524ecd9d397f18ae757850 (DEFAULT_TABLE table=test-table [id=834f7a112846405e9fe07a09cf9b4b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:55.597604 6015 tablet_service.cc:1467] Processing CreateTablet for tablet 573a07d8a0524ecd9d397f18ae757850 (DEFAULT_TABLE table=test-table [id=834f7a112846405e9fe07a09cf9b4b6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:40:55.598171 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573a07d8a0524ecd9d397f18ae757850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:55.598925 6146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573a07d8a0524ecd9d397f18ae757850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:55.598934 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573a07d8a0524ecd9d397f18ae757850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:55.609428 6359 tablet_bootstrap.cc:492] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Bootstrap starting.
I20260927 22:40:55.613401 6359 tablet_bootstrap.cc:654] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:55.613556 6361 tablet_bootstrap.cc:492] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Bootstrap starting.
I20260927 22:40:55.614955 6359 log.cc:824] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:55.617261 6361 tablet_bootstrap.cc:654] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:55.617604 6360 tablet_bootstrap.cc:492] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Bootstrap starting.
I20260927 22:40:55.618553 6359 tablet_bootstrap.cc:492] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: No bootstrap required, opened a new log
I20260927 22:40:55.618561 6361 log.cc:824] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:55.618842 6359 ts_tablet_manager.cc:1397] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:40:55.621104 6360 tablet_bootstrap.cc:654] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:55.621732 6361 tablet_bootstrap.cc:492] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: No bootstrap required, opened a new log
I20260927 22:40:55.622001 6361 ts_tablet_manager.cc:1397] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:40:55.622314 6360 log.cc:824] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:55.625299 6360 tablet_bootstrap.cc:492] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: No bootstrap required, opened a new log
I20260927 22:40:55.625550 6360 ts_tablet_manager.cc:1397] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 22:40:55.627961 6359 raft_consensus.cc:348] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.628345 6359 raft_consensus.cc:374] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:55.628463 6359 raft_consensus.cc:729] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c13ea283e14ca09b6d25694c50d2c0, State: Initialized, Role: FOLLOWER
I20260927 22:40:55.629052 6359 consensus_queue.cc:260] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.630098 6341 heartbeater.cc:503] Master 127.1.53.126:34641 was elected leader, sending a full tablet report...
I20260927 22:40:55.631901 6359 ts_tablet_manager.cc:1428] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:40:55.632371 6361 raft_consensus.cc:348] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.632776 6361 raft_consensus.cc:374] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:55.632860 6361 raft_consensus.cc:729] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a37e8b740f4e1d8c804377c4400d12, State: Initialized, Role: FOLLOWER
I20260927 22:40:55.634029 6361 consensus_queue.cc:260] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.635116 6211 heartbeater.cc:503] Master 127.1.53.126:34641 was elected leader, sending a full tablet report...
I20260927 22:40:55.635785 6360 raft_consensus.cc:348] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.636289 6360 raft_consensus.cc:374] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:55.636404 6360 raft_consensus.cc:729] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9d6eb070374571ad4f6890c3256fed, State: Initialized, Role: FOLLOWER
I20260927 22:40:55.636998 6360 consensus_queue.cc:260] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.638082 6361 ts_tablet_manager.cc:1428] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 22:40:55.638298 6080 heartbeater.cc:503] Master 127.1.53.126:34641 was elected leader, sending a full tablet report...
I20260927 22:40:55.638945 6360 ts_tablet_manager.cc:1428] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260927 22:40:55.647158 6081 tablet.cc:2369] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:40:55.655812 6367 raft_consensus.cc:482] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:40:55.656136 6367 raft_consensus.cc:504] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.658267 6367 leader_election.cc:288] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91c13ea283e14ca09b6d25694c50d2c0 (127.1.53.67:39377), 98a37e8b740f4e1d8c804377c4400d12 (127.1.53.66:37417)
I20260927 22:40:55.661754 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.658773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:58355 (local address 127.1.53.67:39377)
0927 22:40:55.659067 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.659076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.659734 (+ 658us) server_negotiation.cc:407] Connection header received
0927 22:40:55.659938 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.659943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.660002 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.660096 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.661132 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.661138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.661143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.661160 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.661193 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.661204 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.661328 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.661404 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.661491 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 22:40:55.661856 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.658830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:45599 (local address 127.1.53.66:37417)
0927 22:40:55.659243 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.659251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.659972 (+ 721us) server_negotiation.cc:407] Connection header received
0927 22:40:55.660133 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.660137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.660183 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.660316 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.661256 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.661267 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.661271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.661286 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.661314 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.661326 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.661446 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.661537 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.661597 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260927 22:40:55.662101 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.658749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:37417 (local address 127.1.53.65:45599)
0927 22:40:55.659872 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.659886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.660009 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.660343 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.660348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.660374 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.661071 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.661073 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.661084 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.661116 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.661433 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.661438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.661442 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.661598 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.661642 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":476,"threads_started":1}
I20260927 22:40:55.662317 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.658583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39377 (local address 127.1.53.65:58355)
0927 22:40:55.659617 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.659640 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:55.659788 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:55.660118 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:55.660123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:55.660146 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:40:55.660898 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:40:55.660903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:40:55.660923 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:40:55.660984 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:40:55.661294 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:40:55.661298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:40:55.661304 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:40:55.661445 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:40:55.661491 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":890,"threads_started":1}
I20260927 22:40:55.663142 6166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573a07d8a0524ecd9d397f18ae757850" candidate_uuid: "4a9d6eb070374571ad4f6890c3256fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a37e8b740f4e1d8c804377c4400d12" is_pre_election: true
I20260927 22:40:55.663406 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573a07d8a0524ecd9d397f18ae757850" candidate_uuid: "4a9d6eb070374571ad4f6890c3256fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c13ea283e14ca09b6d25694c50d2c0" is_pre_election: true
I20260927 22:40:55.663765 6166 raft_consensus.cc:2454] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a9d6eb070374571ad4f6890c3256fed in term 0.
I20260927 22:40:55.663910 6296 raft_consensus.cc:2454] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a9d6eb070374571ad4f6890c3256fed in term 0.
I20260927 22:40:55.664512 5968 leader_election.cc:302] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9d6eb070374571ad4f6890c3256fed, 91c13ea283e14ca09b6d25694c50d2c0; no voters:
I20260927 22:40:55.664887 6367 raft_consensus.cc:2781] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:40:55.665012 6367 raft_consensus.cc:482] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:40:55.665124 6367 raft_consensus.cc:3037] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:55.667423 6367 raft_consensus.cc:504] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.668045 6367 leader_election.cc:288] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [CANDIDATE]: Term 1 election: Requested vote from peers 91c13ea283e14ca09b6d25694c50d2c0 (127.1.53.67:39377), 98a37e8b740f4e1d8c804377c4400d12 (127.1.53.66:37417)
I20260927 22:40:55.668553 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573a07d8a0524ecd9d397f18ae757850" candidate_uuid: "4a9d6eb070374571ad4f6890c3256fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c13ea283e14ca09b6d25694c50d2c0"
I20260927 22:40:55.668581 6166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573a07d8a0524ecd9d397f18ae757850" candidate_uuid: "4a9d6eb070374571ad4f6890c3256fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a37e8b740f4e1d8c804377c4400d12"
I20260927 22:40:55.668776 6166 raft_consensus.cc:3037] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:55.668777 6296 raft_consensus.cc:3037] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:55.672092 6296 raft_consensus.cc:2454] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a9d6eb070374571ad4f6890c3256fed in term 1.
I20260927 22:40:55.672093 6166 raft_consensus.cc:2454] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a9d6eb070374571ad4f6890c3256fed in term 1.
I20260927 22:40:55.672585 5968 leader_election.cc:302] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9d6eb070374571ad4f6890c3256fed, 91c13ea283e14ca09b6d25694c50d2c0; no voters:
I20260927 22:40:55.672962 6367 raft_consensus.cc:2781] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:55.673512 6367 raft_consensus.cc:686] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 1 LEADER]: Becoming Leader. State: Replica: 4a9d6eb070374571ad4f6890c3256fed, State: Running, Role: LEADER
I20260927 22:40:55.674017 6367 consensus_queue.cc:237] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } }
I20260927 22:40:55.678762 5894 catalog_manager.cc:5495] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a9d6eb070374571ad4f6890c3256fed (127.1.53.65). New cstate: current_term: 1 leader_uuid: "4a9d6eb070374571ad4f6890c3256fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a9d6eb070374571ad4f6890c3256fed" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 } health_report { overall_health: UNKNOWN } } }
I20260927 22:40:55.702808 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.700332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.1.53.65:45665)
0927 22:40:55.700565 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.700572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.700715 (+ 143us) server_negotiation.cc:407] Connection header received
0927 22:40:55.701031 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.701037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.701121 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.701243 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.702149 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.702154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.702160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.702182 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.702214 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.702225 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.702348 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.702461 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.702550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:40:55.711529 6166 raft_consensus.cc:1261] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 1 FOLLOWER]: Refusing update from remote peer 4a9d6eb070374571ad4f6890c3256fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:55.711527 6296 raft_consensus.cc:1261] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 1 FOLLOWER]: Refusing update from remote peer 4a9d6eb070374571ad4f6890c3256fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:40:55.712766 6370 consensus_queue.cc:1035] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c13ea283e14ca09b6d25694c50d2c0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:55.713191 6367 consensus_queue.cc:1035] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a37e8b740f4e1d8c804377c4400d12" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:40:55.724309 6373 mvcc.cc:205] Tried to move back new op lower bound from 7334088112983158784 to 7334088112844468224. Current Snapshot: MvccSnapshot[applied={T|T < 7334088112983158784}]
I20260927 22:40:55.730125 6374 mvcc.cc:205] Tried to move back new op lower bound from 7334088112983158784 to 7334088112844468224. Current Snapshot: MvccSnapshot[applied={T|T < 7334088112983158784}]
I20260927 22:40:55.737253 5894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:55.737546 5894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:40:55.741285 5894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09c01c34d5e148f3a2ac8af20ab05c7d: Sending DeleteTablet for 3 replicas of tablet 573a07d8a0524ecd9d397f18ae757850
I20260927 22:40:55.742468 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 573a07d8a0524ecd9d397f18ae757850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:55 UTC) from {username='slave'} at 127.0.0.1:46906
I20260927 22:40:55.742468 6146 tablet_service.cc:1514] Processing DeleteTablet for tablet 573a07d8a0524ecd9d397f18ae757850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:55 UTC) from {username='slave'} at 127.0.0.1:58994
I20260927 22:40:55.742623 6015 tablet_service.cc:1514] Processing DeleteTablet for tablet 573a07d8a0524ecd9d397f18ae757850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:40:55 UTC) from {username='slave'} at 127.0.0.1:43794
I20260927 22:40:55.747462 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5953
I20260927 22:40:55.752729 6384 tablet_replica.cc:330] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: stopping tablet replica
I20260927 22:40:55.753314 6384 raft_consensus.cc:2229] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:40:55.754101 6384 raft_consensus.cc:2258] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:55.754973 6382 tablet_replica.cc:330] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: stopping tablet replica
I20260927 22:40:55.755511 6382 raft_consensus.cc:2229] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:55.756131 6382 raft_consensus.cc:2258] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:55.758284 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.748768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.1.53.65:45665)
0927 22:40:55.748990 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.748999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.755326 (+ 6327us) server_negotiation.cc:407] Connection header received
0927 22:40:55.755707 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.755715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.755799 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.755887 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.757480 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.757485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.757490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.757510 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.757549 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.757564 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.757661 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.757911 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.757969 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:40:55.758533 6384 ts_tablet_manager.cc:1905] T 573a07d8a0524ecd9d397f18ae757850 P 4a9d6eb070374571ad4f6890c3256fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:55.758963 6383 tablet_replica.cc:330] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: stopping tablet replica
I20260927 22:40:55.759315 6075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:55.759399 6383 raft_consensus.cc:2229] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:40:55.759824 6382 ts_tablet_manager.cc:1905] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:55.759965 6383 raft_consensus.cc:2258] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:40:55.762401 6383 ts_tablet_manager.cc:1905] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:40:55.766502 6382 ts_tablet_manager.cc:1918] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:55.766674 6382 log.cc:1196] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/573a07d8a0524ecd9d397f18ae757850
I20260927 22:40:55.767195 6382 ts_tablet_manager.cc:1939] T 573a07d8a0524ecd9d397f18ae757850 P 98a37e8b740f4e1d8c804377c4400d12: Deleting consensus metadata
I20260927 22:40:55.768347 6383 ts_tablet_manager.cc:1918] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:40:55.768510 6383 log.cc:1196] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/573a07d8a0524ecd9d397f18ae757850
I20260927 22:40:55.769021 6383 ts_tablet_manager.cc:1939] T 573a07d8a0524ecd9d397f18ae757850 P 91c13ea283e14ca09b6d25694c50d2c0: Deleting consensus metadata
I20260927 22:40:55.769217 5879 catalog_manager.cc:4841] TS 98a37e8b740f4e1d8c804377c4400d12 (127.1.53.66:37417): tablet 573a07d8a0524ecd9d397f18ae757850 (table test-table [id=834f7a112846405e9fe07a09cf9b4b6b]) successfully deleted
I20260927 22:40:55.770479 5878 catalog_manager.cc:4841] TS 91c13ea283e14ca09b6d25694c50d2c0 (127.1.53.67:39377): tablet 573a07d8a0524ecd9d397f18ae757850 (table test-table [id=834f7a112846405e9fe07a09cf9b4b6b]) successfully deleted
I20260927 22:40:55.857302 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5953
I20260927 22:40:55.875862 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6084
W20260927 22:40:55.875839 5878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:45665 (error 108)
I20260927 22:40:55.877764 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.877311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43828)
0927 22:40:55.877491 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.877619 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 22:40:55.878072 5878 catalog_manager.cc:4584] TS 4a9d6eb070374571ad4f6890c3256fed (127.1.53.65:45665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 573a07d8a0524ecd9d397f18ae757850: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
I20260927 22:40:55.879099 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.876701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.1.53.66:37417)
0927 22:40:55.876957 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.876964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.877052 (+ 88us) server_negotiation.cc:407] Connection header received
0927 22:40:55.877279 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.877283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.877370 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.877495 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.878504 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.878509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.878513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.878528 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.878559 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.878570 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.878679 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.878757 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.878801 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 22:40:55.879776 6206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:55.931756 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.931201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43844)
0927 22:40:55.931451 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.931563 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:40:55.932886 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.932503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43860)
0927 22:40:55.932657 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.932761 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:40:55.968611 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6084
I20260927 22:40:55.984701 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6214
I20260927 22:40:55.987565 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.985626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.1.53.67:39377)
0927 22:40:55.985793 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:55.985798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:55.985827 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:40:55.986016 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:55.986020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:55.986116 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:55.986216 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:55.987019 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:55.987024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:55.987027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:55.987042 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:55.987074 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:55.987085 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:55.987200 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:55.987256 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 22:40:55.987300 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:40:55.988353 6336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:55.993491 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.993063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43866)
0927 22:40:55.993238 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.993336 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:40:55.994572 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:55.994247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43876)
0927 22:40:55.994374 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:55.994457 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:40:56.075553 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:56.074939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43882)
0927 22:40:56.075208 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:56.075361 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 22:40:56.077328 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:56.076796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45665 (local address 127.0.0.1:43890)
0927 22:40:56.077049 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:56.077153 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:45665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:40:56.077762 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6214
I20260927 22:40:56.093925 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 5863
I20260927 22:40:56.095176 5924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:56.199865 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 5863
2026-09-27T22:40:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2597 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:40:56.242345 1237 mini_oidc.cc:152] Starting JWKS server
I20260927 22:40:56.244972 1237 webserver.cc:460] Webserver started at https://127.0.0.1:33267/ using document root <none> and password file <none>
I20260927 22:40:56.245299 1237 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:40:56.245951 1237 webserver.cc:460] Webserver started at http://127.0.0.1:44257/ using document root <none> and password file <none>
2026-09-27T22:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:56Z Disabled control of system clock
I20260927 22:40:56.275635 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:35077
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:35077
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33267/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/testchainca.cert with env {}
W20260927 22:40:56.602665 6396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:56.602959 6396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:56.603047 6396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:56.610221 6396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:56.610329 6396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:56.610369 6396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:56.610424 6396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:56.610466 6396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:40:56.610525 6396 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:40:56.610564 6396 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33267/jwks.json
I20260927 22:40:56.620877 6396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:35077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:35077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33267/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:56.622190 6396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:56.623525 6396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:56.635833 6401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:56.635758 6402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:56.635777 6404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:56.636751 6396 server_base.cc:1034] running on GCE node
I20260927 22:40:56.637568 6396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:56.638945 6396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:56.640189 6396 hybrid_clock.cc:648] HybridClock initialized: now 1790548856640137 us; error 64 us; skew 500 ppm
I20260927 22:40:56.640558 6396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:56.642786 6396 webserver.cc:460] Webserver started at http://127.1.53.126:35147/ using document root <none> and password file <none>
I20260927 22:40:56.643447 6396 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:56.643533 6396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:56.643783 6396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:56.646724 6396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "2404419787d64b8a9ad4297afdb6f81f"
format_stamp: "Formatted at 2026-09-27 22:40:56 on dist-test-slave-pp80"
server_key: "aca4e3361fb673981eb8bf6b57b015e1"
server_key_iv: "543f4d467e80dc2c21f9ee5eea9e7a59"
server_key_version: "encryptionkey@0"
I20260927 22:40:56.647508 6396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "2404419787d64b8a9ad4297afdb6f81f"
format_stamp: "Formatted at 2026-09-27 22:40:56 on dist-test-slave-pp80"
server_key: "aca4e3361fb673981eb8bf6b57b015e1"
server_key_iv: "543f4d467e80dc2c21f9ee5eea9e7a59"
server_key_version: "encryptionkey@0"
I20260927 22:40:56.652400 6396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:40:56.655610 6410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:56.656965 6396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:56.657104 6396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "2404419787d64b8a9ad4297afdb6f81f"
format_stamp: "Formatted at 2026-09-27 22:40:56 on dist-test-slave-pp80"
server_key: "aca4e3361fb673981eb8bf6b57b015e1"
server_key_iv: "543f4d467e80dc2c21f9ee5eea9e7a59"
server_key_version: "encryptionkey@0"
I20260927 22:40:56.657267 6396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:56.686189 6396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:56.687058 6396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:56.687300 6396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:56.705951 6396 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:35077
I20260927 22:40:56.705996 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:35077 every 8 connection(s)
I20260927 22:40:56.707381 6396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:56.711637 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:56.716230 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6396
I20260927 22:40:56.716537 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:56.717149 1237 external_mini_cluster.cc:1428] Setting key 868ec91c359c59b2349295417d9a3fcb
I20260927 22:40:56.722366 6462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f: Bootstrap starting.
I20260927 22:40:56.725986 6462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:56.727191 6462 log.cc:824] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:56.730458 6462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f: No bootstrap required, opened a new log
I20260927 22:40:56.731588 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:56.719128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.1.53.126:35077)
0927 22:40:56.720127 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:56.720141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:56.720186 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:40:56.721849 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:56.721872 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:56.722318 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:56.722683 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:56.723296 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:56.724643 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:56.726541 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:56.727580 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:56.727842 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:56.729104 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:56.729112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:56.729130 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:56.729663 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:56.729736 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:56.729760 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:56.729870 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:56.730148 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:40:56.730590 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":371,"threads_started":1}
I20260927 22:40:56.735719 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:40:56.738292 6462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404419787d64b8a9ad4297afdb6f81f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35077 } }
I20260927 22:40:56.738647 6462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:56.738770 6462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2404419787d64b8a9ad4297afdb6f81f, State: Initialized, Role: FOLLOWER
I20260927 22:40:56.739370 6462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404419787d64b8a9ad4297afdb6f81f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35077 } }
I20260927 22:40:56.739589 6462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:56.739733 6462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:56.739892 6462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:56.742071 6462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404419787d64b8a9ad4297afdb6f81f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35077 } }
I20260927 22:40:56.742621 6462 leader_election.cc:302] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2404419787d64b8a9ad4297afdb6f81f; no voters:
I20260927 22:40:56.743199 6462 leader_election.cc:288] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:56.743351 6467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:56.744083 6467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [term 1 LEADER]: Becoming Leader. State: Replica: 2404419787d64b8a9ad4297afdb6f81f, State: Running, Role: LEADER
I20260927 22:40:56.744629 6467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404419787d64b8a9ad4297afdb6f81f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35077 } }
I20260927 22:40:56.745354 6462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:56.747674 6468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2404419787d64b8a9ad4297afdb6f81f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404419787d64b8a9ad4297afdb6f81f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35077 } } }
I20260927 22:40:56.747644 6469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2404419787d64b8a9ad4297afdb6f81f. Latest consensus state: current_term: 1 leader_uuid: "2404419787d64b8a9ad4297afdb6f81f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404419787d64b8a9ad4297afdb6f81f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35077 } } }
I20260927 22:40:56.752492 6468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:56.752665 6469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:56.756767 6473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:56.762952 6473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:56.771762 6473 catalog_manager.cc:1347] Generated new cluster ID: 41c86a25e80948859cd1f92bd0638b82
I20260927 22:40:56.771953 6473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:56.780146 6473 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:56.781512 6473 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:56.794698 6473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2404419787d64b8a9ad4297afdb6f81f: Generated new TSK 0
I20260927 22:40:56.795601 6473 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:40:56.844074 6492 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:40:56.844291 6492 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:40:56.845407 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:56.823032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.1.53.126:35077)
0927 22:40:56.823194 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:56.823199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:56.823655 (+ 456us) server_negotiation.cc:407] Connection header received
0927 22:40:56.823948 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:56.823954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:56.824052 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:56.824215 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:40:56.824781 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:56.825944 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:56.827551 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:56.828271 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:56.828408 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:56.828919 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:40:56.844717 (+ 15798us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:40:56.845111 (+ 394us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55342: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 22:40:56.845547 6465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55342: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:40:56.847182 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:40:56.822893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35077 (local address 127.0.0.1:55342)
0927 22:40:56.823545 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:56.823565 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:56.823715 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:56.824308 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:56.824312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:56.824323 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:40:56.824637 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:56.824645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:56.826109 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:56.826113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:56.827404 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:56.827415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:56.828463 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:56.828468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:56.828591 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:56.828647 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:40:56.845080 (+ 16433us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:40:56.845493 (+ 413us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.53.126:35077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":328,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.53.126:35077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:40:56.848655 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6396
I20260927 22:40:56.849902 6457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:56.942058 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6396
2026-09-27T22:40:56Z chronyd exiting
I20260927 22:40:56.992426 1237 test_util.cc:175] -----------------------------------------------
I20260927 22:40:56.992575 1237 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790548812741971-1237-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:40:56.996394 1237 mini_oidc.cc:152] Starting JWKS server
I20260927 22:40:56.998173 1237 webserver.cc:460] Webserver started at https://127.0.0.1:40643/ using document root <none> and password file <none>
I20260927 22:40:56.998420 1237 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:40:56.999066 1237 webserver.cc:460] Webserver started at http://127.0.0.1:40707/ using document root <none> and password file <none>
2026-09-27T22:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:57Z Disabled control of system clock
I20260927 22:40:57.026194 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42821
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42821
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40643/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/testchainca.cert with env {}
W20260927 22:40:57.351321 6499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:57.351621 6499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:57.351708 6499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:57.358976 6499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:57.359086 6499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:57.359124 6499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:57.359180 6499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:57.359216 6499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:40:57.359277 6499 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:40:57.359313 6499 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40643/jwks.json
I20260927 22:40:57.369205 6499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40643/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:57.370450 6499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:57.371740 6499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:57.383399 6505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:57.383639 6504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:57.383608 6507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:57.384855 6499 server_base.cc:1034] running on GCE node
I20260927 22:40:57.385641 6499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:57.387081 6499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:57.388329 6499 hybrid_clock.cc:648] HybridClock initialized: now 1790548857388266 us; error 57 us; skew 500 ppm
I20260927 22:40:57.388633 6499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:57.391047 6499 webserver.cc:460] Webserver started at http://127.1.53.126:41293/ using document root <none> and password file <none>
I20260927 22:40:57.391709 6499 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:57.391801 6499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:57.392073 6499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:57.395208 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "88e3102823b049b297933af4c1d70365"
format_stamp: "Formatted at 2026-09-27 22:40:57 on dist-test-slave-pp80"
server_key: "f44e1ad926a7925ae6400c8f3536ce53"
server_key_iv: "9427a5474156dab453262afbbd26d046"
server_key_version: "encryptionkey@0"
I20260927 22:40:57.395928 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "88e3102823b049b297933af4c1d70365"
format_stamp: "Formatted at 2026-09-27 22:40:57 on dist-test-slave-pp80"
server_key: "f44e1ad926a7925ae6400c8f3536ce53"
server_key_iv: "9427a5474156dab453262afbbd26d046"
server_key_version: "encryptionkey@0"
I20260927 22:40:57.401024 6499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:40:57.404340 6513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:57.405707 6499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:40:57.405884 6499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "88e3102823b049b297933af4c1d70365"
format_stamp: "Formatted at 2026-09-27 22:40:57 on dist-test-slave-pp80"
server_key: "f44e1ad926a7925ae6400c8f3536ce53"
server_key_iv: "9427a5474156dab453262afbbd26d046"
server_key_version: "encryptionkey@0"
I20260927 22:40:57.406121 6499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:57.418488 6499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:57.419267 6499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:57.419548 6499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:57.437515 6499 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42821
I20260927 22:40:57.437541 6564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42821 every 8 connection(s)
I20260927 22:40:57.438885 6499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:57.442720 6565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:57.445770 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6499
I20260927 22:40:57.445963 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:57.446593 1237 external_mini_cluster.cc:1428] Setting key de6430f30c8db870cc6a26a51f1ce479
I20260927 22:40:57.454968 6565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365: Bootstrap starting.
I20260927 22:40:57.458654 6565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:57.459723 6565 log.cc:824] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:57.460136 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:57.448495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.1.53.126:42821)
0927 22:40:57.449485 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:57.449496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:57.449536 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:40:57.451087 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:57.451113 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:57.451605 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:57.451932 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:57.452587 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:57.453836 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:57.455624 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:57.456377 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:57.456515 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:57.457788 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:57.457796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:57.457809 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:57.458324 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:57.458408 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:57.458427 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:57.458514 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:57.458810 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:40:57.459165 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":353,"threads_started":1}
I20260927 22:40:57.462570 6565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365: No bootstrap required, opened a new log
I20260927 22:40:57.463846 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:40:57.470302 6565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3102823b049b297933af4c1d70365" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42821 } }
I20260927 22:40:57.470808 6565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:57.470948 6565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e3102823b049b297933af4c1d70365, State: Initialized, Role: FOLLOWER
I20260927 22:40:57.471596 6565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3102823b049b297933af4c1d70365" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42821 } }
I20260927 22:40:57.471840 6565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:57.471982 6565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:57.472157 6565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:57.474300 6565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3102823b049b297933af4c1d70365" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42821 } }
I20260927 22:40:57.474799 6565 leader_election.cc:302] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e3102823b049b297933af4c1d70365; no voters:
I20260927 22:40:57.475345 6565 leader_election.cc:288] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:57.475474 6570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:57.476267 6570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [term 1 LEADER]: Becoming Leader. State: Replica: 88e3102823b049b297933af4c1d70365, State: Running, Role: LEADER
I20260927 22:40:57.476805 6570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3102823b049b297933af4c1d70365" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42821 } }
I20260927 22:40:57.477517 6565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:57.479297 6571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e3102823b049b297933af4c1d70365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3102823b049b297933af4c1d70365" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42821 } } }
I20260927 22:40:57.479694 6571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:57.480633 6575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:57.480487 6572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e3102823b049b297933af4c1d70365. Latest consensus state: current_term: 1 leader_uuid: "88e3102823b049b297933af4c1d70365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3102823b049b297933af4c1d70365" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42821 } } }
I20260927 22:40:57.481637 6572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:57.487246 6575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:57.496358 6575 catalog_manager.cc:1347] Generated new cluster ID: 286d3a7bb1c04e659bafe235a3c7e277
I20260927 22:40:57.496475 6575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:57.516470 6575 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:57.518291 6575 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:57.536942 6575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e3102823b049b297933af4c1d70365: Generated new TSK 0
I20260927 22:40:57.538152 6575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:40:57.581457 6595 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:40:57.581595 6595 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:40:57.582521 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:57.554652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.1.53.126:42821)
0927 22:40:57.554861 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:57.554868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:57.555246 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:40:57.555536 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:57.555540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:57.555613 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:57.555752 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:40:57.556346 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:57.557083 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:57.559163 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:57.559621 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:57.559723 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:57.560425 (+ 702us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:40:57.581988 (+ 21563us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:40:57.582301 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37338: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:40:57.582669 6568 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37338: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 22:40:57.582962 6594 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:40:57.554448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42821 (local address 127.0.0.1:37338)
0927 22:40:57.555125 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:57.555146 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:57.555308 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:57.555793 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:57.555798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:57.555812 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:40:57.556119 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:57.556131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:57.557222 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:57.557228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:57.558981 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:57.558996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:57.559787 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:57.559794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:57.559996 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:57.560042 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:40:57.582252 (+ 22210us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:40:57.582548 (+ 296us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.53.126:42821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":350,"threads_started":1}
I20260927 22:40:57.584407 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6499
I20260927 22:40:57.585732 6560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:57.689327 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6499
2026-09-27T22:40:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:40:57.747859 1237 mini_oidc.cc:152] Starting JWKS server
I20260927 22:40:57.749749 1237 webserver.cc:460] Webserver started at https://127.0.0.1:33435/ using document root <none> and password file <none>
I20260927 22:40:57.749997 1237 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:40:57.750625 1237 webserver.cc:460] Webserver started at http://127.0.0.1:44179/ using document root <none> and password file <none>
2026-09-27T22:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:57Z Disabled control of system clock
I20260927 22:40:57.779394 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37837
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37837
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33435/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:40:58.107867 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:58.108146 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:58.108232 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:58.115326 6602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:58.115435 6602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:58.115478 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:58.115525 6602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:58.115568 6602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:40:58.115628 6602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:40:58.115666 6602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33435/jwks.json
I20260927 22:40:58.124846 6602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33435/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:58.126132 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:58.127902 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:58.139988 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:58.139911 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:58.139953 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:58.140861 6602 server_base.cc:1034] running on GCE node
I20260927 22:40:58.141703 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:58.143015 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:58.144212 6602 hybrid_clock.cc:648] HybridClock initialized: now 1790548858144174 us; error 25 us; skew 500 ppm
I20260927 22:40:58.144632 6602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:58.146971 6602 webserver.cc:460] Webserver started at http://127.1.53.126:41533/ using document root <none> and password file <none>
I20260927 22:40:58.147648 6602 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:58.147760 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:58.148089 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:58.151119 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "f8d670bc87af44a69a16064facf96d57"
format_stamp: "Formatted at 2026-09-27 22:40:58 on dist-test-slave-pp80"
server_key: "64a207caf7e9dc9c26040eb47bf57fb3"
server_key_iv: "18fff8a0cdde0bf295d7cfaab9222790"
server_key_version: "encryptionkey@0"
I20260927 22:40:58.151947 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "f8d670bc87af44a69a16064facf96d57"
format_stamp: "Formatted at 2026-09-27 22:40:58 on dist-test-slave-pp80"
server_key: "64a207caf7e9dc9c26040eb47bf57fb3"
server_key_iv: "18fff8a0cdde0bf295d7cfaab9222790"
server_key_version: "encryptionkey@0"
I20260927 22:40:58.157063 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:40:58.160420 6616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:58.161854 6602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:40:58.162061 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "f8d670bc87af44a69a16064facf96d57"
format_stamp: "Formatted at 2026-09-27 22:40:58 on dist-test-slave-pp80"
server_key: "64a207caf7e9dc9c26040eb47bf57fb3"
server_key_iv: "18fff8a0cdde0bf295d7cfaab9222790"
server_key_version: "encryptionkey@0"
I20260927 22:40:58.162281 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:58.186080 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:58.186942 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:58.187235 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:58.205363 6602 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37837
I20260927 22:40:58.205371 6667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37837 every 8 connection(s)
I20260927 22:40:58.206754 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:58.209466 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6602
I20260927 22:40:58.209700 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:58.210474 1237 external_mini_cluster.cc:1428] Setting key 4e882de0ddc3f6b60c2e249e51df5599
I20260927 22:40:58.211323 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:58.224602 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:58.212532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.1.53.126:37837)
0927 22:40:58.213520 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:58.213531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:58.213572 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:40:58.215713 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:58.215735 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:58.216205 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:58.216626 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:58.217189 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.218507 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.220328 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.220986 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.221116 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:58.222289 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:58.222296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:58.222311 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:58.222871 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:58.222945 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:58.222969 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:58.223058 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:58.223278 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:40:58.223613 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":364,"threads_started":1}
I20260927 22:40:58.224705 6668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57: Bootstrap starting.
I20260927 22:40:58.227828 6668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:58.228297 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:40:58.229059 6668 log.cc:824] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:58.231950 6668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57: No bootstrap required, opened a new log
I20260927 22:40:58.238693 6668 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d670bc87af44a69a16064facf96d57" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37837 } }
I20260927 22:40:58.239060 6668 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:58.239166 6668 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d670bc87af44a69a16064facf96d57, State: Initialized, Role: FOLLOWER
I20260927 22:40:58.239696 6668 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d670bc87af44a69a16064facf96d57" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37837 } }
I20260927 22:40:58.239924 6668 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:58.240073 6668 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:58.240299 6668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:58.242506 6668 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d670bc87af44a69a16064facf96d57" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37837 } }
I20260927 22:40:58.242993 6668 leader_election.cc:302] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8d670bc87af44a69a16064facf96d57; no voters:
I20260927 22:40:58.243561 6668 leader_election.cc:288] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:58.243762 6673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:58.244515 6673 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [term 1 LEADER]: Becoming Leader. State: Replica: f8d670bc87af44a69a16064facf96d57, State: Running, Role: LEADER
I20260927 22:40:58.245517 6668 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:58.245455 6673 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d670bc87af44a69a16064facf96d57" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37837 } }
I20260927 22:40:58.248198 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8d670bc87af44a69a16064facf96d57. Latest consensus state: current_term: 1 leader_uuid: "f8d670bc87af44a69a16064facf96d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d670bc87af44a69a16064facf96d57" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37837 } } }
I20260927 22:40:58.248625 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:58.249197 6674 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8d670bc87af44a69a16064facf96d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d670bc87af44a69a16064facf96d57" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37837 } } }
I20260927 22:40:58.249527 6674 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57 [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:58.249770 6682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:58.256446 6682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:58.271358 6682 catalog_manager.cc:1347] Generated new cluster ID: 318e39d094d441e0a0a930c607727b88
I20260927 22:40:58.271489 6682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:58.297353 6682 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:58.298671 6682 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:58.316032 6682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8d670bc87af44a69a16064facf96d57: Generated new TSK 0
I20260927 22:40:58.316926 6682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:40:58.345767 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:58.336805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.1.53.126:37837)
0927 22:40:58.337023 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:58.337030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:58.337336 (+ 306us) server_negotiation.cc:407] Connection header received
0927 22:40:58.337559 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:58.337566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:58.337671 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:58.337877 (+ 206us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:40:58.338294 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.339533 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.340958 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.341731 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.341893 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:58.342374 (+ 481us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:40:58.345041 (+ 2667us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:40:58.345395 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 22:40:58.345927 6671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 22:40:58.347595 6697 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:40:58.336635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37837 (local address 127.0.0.1:35284)
0927 22:40:58.337235 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:58.337251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:58.337371 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:58.337901 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:58.337904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:58.337914 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:40:58.338154 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:58.338161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.339693 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.339697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:58.340815 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:58.340826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.341933 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.341937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:58.342054 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:58.342085 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:40:58.345255 (+ 3170us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:40:58.347190 (+ 1935us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.53.126:37837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33435/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":1643,"thread_start_us":328,"threads_started":1}
I20260927 22:40:58.348865 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6602
I20260927 22:40:58.350097 6663 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:40:58.361611 6698 webserver.cc:533] Webserver: sslize failed:
E20260927 22:40:58.361739 6698 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:40:58.444563 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6602
2026-09-27T22:40:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:40:58Z Disabled control of system clock
I20260927 22:40:58.526775 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:40999
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:40999
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/testchainca.cert with env {}
W20260927 22:40:58.854035 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:40:58.854447 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:40:58.854542 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:40:58.861757 6703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:40:58.861874 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:40:58.861912 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:40:58.861966 6703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:40:58.862010 6703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:40:58.862066 6703 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:40:58.862103 6703 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:40:58.871954 6703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:40999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:40999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:40:58.873288 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:40:58.874689 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:40:58.886564 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:58.886499 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:40:58.886535 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:40:58.887277 6703 server_base.cc:1034] running on GCE node
I20260927 22:40:58.888118 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:40:58.889577 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:40:58.890829 6703 hybrid_clock.cc:648] HybridClock initialized: now 1790548858890779 us; error 59 us; skew 500 ppm
I20260927 22:40:58.891220 6703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:40:58.893646 6703 webserver.cc:460] Webserver started at http://127.1.53.126:38965/ using document root <none> and password file <none>
I20260927 22:40:58.894372 6703 fs_manager.cc:359] Metadata directory not provided
I20260927 22:40:58.894513 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:40:58.894841 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:40:58.897753 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "980a6887c66e4b0ab7064c2834eae43f"
format_stamp: "Formatted at 2026-09-27 22:40:58 on dist-test-slave-pp80"
server_key: "799df905a2c55e89600a638f66dddf4f"
server_key_iv: "b37893720aa4dd920bd34d9c0ab87416"
server_key_version: "encryptionkey@0"
I20260927 22:40:58.898552 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "980a6887c66e4b0ab7064c2834eae43f"
format_stamp: "Formatted at 2026-09-27 22:40:58 on dist-test-slave-pp80"
server_key: "799df905a2c55e89600a638f66dddf4f"
server_key_iv: "b37893720aa4dd920bd34d9c0ab87416"
server_key_version: "encryptionkey@0"
I20260927 22:40:58.903635 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:40:58.907220 6717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:40:58.908747 6703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:40:58.909020 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "980a6887c66e4b0ab7064c2834eae43f"
format_stamp: "Formatted at 2026-09-27 22:40:58 on dist-test-slave-pp80"
server_key: "799df905a2c55e89600a638f66dddf4f"
server_key_iv: "b37893720aa4dd920bd34d9c0ab87416"
server_key_version: "encryptionkey@0"
I20260927 22:40:58.909269 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:40:58.940828 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:40:58.941773 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:40:58.942106 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:40:58.961565 6703 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:40999
I20260927 22:40:58.961587 6768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:40999 every 8 connection(s)
I20260927 22:40:58.963040 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:40:58.966533 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6703
I20260927 22:40:58.966723 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:40:58.967439 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:40:58.967555 1237 external_mini_cluster.cc:1428] Setting key 53b7d32f88ef74a34a2049a54cf7f565
I20260927 22:40:58.981536 6769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f: Bootstrap starting.
I20260927 22:40:58.982314 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:58.969584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.1.53.126:40999)
0927 22:40:58.970728 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:58.970739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:58.970786 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:40:58.972737 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:58.972763 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:58.973301 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:58.973695 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:40:58.974333 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.975741 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.977601 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:58.978322 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:58.978484 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:58.979768 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:40:58.979776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:40:58.979790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:40:58.980415 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:40:58.980509 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:40:58.980530 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:40:58.980626 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:40:58.980848 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:40:58.981190 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":404,"threads_started":1}
I20260927 22:40:58.984875 6769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f: Neither blocks nor log segments found. Creating new log.
I20260927 22:40:58.986068 6769 log.cc:824] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:40:58.986089 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:40:58.989065 6769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f: No bootstrap required, opened a new log
I20260927 22:40:58.996201 6769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980a6887c66e4b0ab7064c2834eae43f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40999 } }
I20260927 22:40:58.996619 6769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:40:58.996717 6769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980a6887c66e4b0ab7064c2834eae43f, State: Initialized, Role: FOLLOWER
I20260927 22:40:58.997328 6769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980a6887c66e4b0ab7064c2834eae43f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40999 } }
I20260927 22:40:58.997550 6769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:40:58.997696 6769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:40:58.997865 6769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:40:59.000133 6769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980a6887c66e4b0ab7064c2834eae43f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40999 } }
I20260927 22:40:59.000646 6769 leader_election.cc:302] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980a6887c66e4b0ab7064c2834eae43f; no voters:
I20260927 22:40:59.001251 6769 leader_election.cc:288] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:40:59.001407 6774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:40:59.002110 6774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [term 1 LEADER]: Becoming Leader. State: Replica: 980a6887c66e4b0ab7064c2834eae43f, State: Running, Role: LEADER
I20260927 22:40:59.002697 6774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980a6887c66e4b0ab7064c2834eae43f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40999 } }
I20260927 22:40:59.003377 6769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:40:59.005741 6776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980a6887c66e4b0ab7064c2834eae43f. Latest consensus state: current_term: 1 leader_uuid: "980a6887c66e4b0ab7064c2834eae43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980a6887c66e4b0ab7064c2834eae43f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40999 } } }
I20260927 22:40:59.005731 6775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "980a6887c66e4b0ab7064c2834eae43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980a6887c66e4b0ab7064c2834eae43f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40999 } } }
I20260927 22:40:59.006158 6776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:59.006233 6775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f [sys.catalog]: This master's current role is: LEADER
I20260927 22:40:59.007689 6779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:40:59.014415 6779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:40:59.024878 6779 catalog_manager.cc:1347] Generated new cluster ID: bf7e4511a5044cd2ad0550894ffadb78
I20260927 22:40:59.025012 6779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:40:59.043361 6779 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:40:59.045262 6779 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:40:59.057761 6779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980a6887c66e4b0ab7064c2834eae43f: Generated new TSK 0
I20260927 22:40:59.058844 6779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:40:59.530277 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:40:59.083770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.1.53.126:40999)
0927 22:40:59.084017 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:40:59.084026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:40:59.084470 (+ 444us) server_negotiation.cc:407] Connection header received
0927 22:40:59.084743 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:40:59.084748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:40:59.084814 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:40:59.084994 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:40:59.085483 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:59.086208 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:59.087575 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:40:59.088021 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:40:59.088169 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:59.088681 (+ 512us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:40:59.529723 (+441042us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:40:59.530071 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56584: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 22:40:59.530424 6772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56584: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:40:59.530675 6798 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:40:59.083715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:40999 (local address 127.0.0.1:56584)
0927 22:40:59.084358 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 22:40:59.084374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:40:59.084547 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:40:59.085019 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:40:59.085023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:40:59.085036 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:40:59.085337 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:59.085346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:59.086349 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:59.086355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:59.087434 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:40:59.087445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:40:59.088218 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:40:59.088228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:40:59.088387 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:40:59.088439 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:40:59.530029 (+441590us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:40:59.530277 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.53.126:40999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":340,"threads_started":1}
I20260927 22:40:59.532488 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6703
I20260927 22:40:59.533663 6764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:40:59.634470 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6703
2026-09-27T22:40:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1171 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (408 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (421 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:00 dist-test-slave-pp80 krb5kdc[6813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:00 dist-test-slave-pp80 krb5kdc[6813](info): set up 2 sockets
Sep 27 22:41:00 dist-test-slave-pp80 krb5kdc[6813](info): commencing operation
krb5kdc: starting...
W20260927 22:41:01.544973 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:01 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:41:01.753714 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46331
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:02.074512 6826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:02.074788 6826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:02.074887 6826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:02.081946 6826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:02.082067 6826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:02.082113 6826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:02.082171 6826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:02.082216 6826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:02.092100 6826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:02.093351 6826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:02.094573 6826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:02.105835 6831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:02.105782 6832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:02.105959 6834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:02.107003 6826 server_base.cc:1034] running on GCE node
I20260927 22:41:02.107723 6826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:02.109086 6826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:02.110278 6826 hybrid_clock.cc:648] HybridClock initialized: now 1790548862110239 us; error 30 us; skew 500 ppm
I20260927 22:41:02.110602 6826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:02 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:02.113612 6826 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:41:02.115046 6826 webserver.cc:460] Webserver started at http://127.1.53.126:35905/ using document root <none> and password file <none>
I20260927 22:41:02.115696 6826 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:02.115818 6826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:02.116145 6826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:02.119046 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "b9f6813ff604460b885b47e5188e2ba3"
format_stamp: "Formatted at 2026-09-27 22:41:02 on dist-test-slave-pp80"
server_key: "9c964826135b7a543d1559e457ed2d0d"
server_key_iv: "2c655c23cb6fb623bffce94518779d07"
server_key_version: "encryptionkey@0"
I20260927 22:41:02.119791 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "b9f6813ff604460b885b47e5188e2ba3"
format_stamp: "Formatted at 2026-09-27 22:41:02 on dist-test-slave-pp80"
server_key: "9c964826135b7a543d1559e457ed2d0d"
server_key_iv: "2c655c23cb6fb623bffce94518779d07"
server_key_version: "encryptionkey@0"
I20260927 22:41:02.124764 6826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:41:02.128082 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:02.129463 6826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:02.129637 6826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "b9f6813ff604460b885b47e5188e2ba3"
format_stamp: "Formatted at 2026-09-27 22:41:02 on dist-test-slave-pp80"
server_key: "9c964826135b7a543d1559e457ed2d0d"
server_key_iv: "2c655c23cb6fb623bffce94518779d07"
server_key_version: "encryptionkey@0"
I20260927 22:41:02.129798 6826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:02.164664 6826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:02.169243 6826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:02.169565 6826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:02.188535 6826 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:46331
I20260927 22:41:02.188575 6892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:46331 every 8 connection(s)
I20260927 22:41:02.189970 6826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:02.194218 6893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:02.194198 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6826
I20260927 22:41:02.194443 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:02.195073 1237 external_mini_cluster.cc:1428] Setting key b6bc620c3971507e173f73ce7dc70727
I20260927 22:41:02.208508 6893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3: Bootstrap starting.
Sep 27 22:41:02 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:02.212565 6893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:02.214066 6893 log.cc:824] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:02.217618 6893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3: No bootstrap required, opened a new log
I20260927 22:41:02.221000 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:02.197070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.1.53.126:46331)
0927 22:41:02.197885 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:02.197897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:02.197938 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:41:02.200446 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:02.200473 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:02.200936 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:02.201309 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:02.202959 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:02.204553 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:02.206418 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:02.207177 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:02.207335 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:02.212286 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:02.212300 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:02.212323 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:02.212382 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:02.216261 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:02.217022 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:02.217031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:02.217039 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:02.217238 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:02.217650 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:02.217656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:02.217659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:02.218389 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:02.218677 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:02.219072 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 22:41:02.219526 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":388,"threads_started":1}
I20260927 22:41:02.226770 6893 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6813ff604460b885b47e5188e2ba3" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46331 } }
I20260927 22:41:02.227101 6893 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:02.227170 6893 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f6813ff604460b885b47e5188e2ba3, State: Initialized, Role: FOLLOWER
I20260927 22:41:02.227715 6893 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6813ff604460b885b47e5188e2ba3" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46331 } }
I20260927 22:41:02.227918 6893 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:02.228019 6893 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:02.228138 6893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:02.230434 6893 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6813ff604460b885b47e5188e2ba3" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46331 } }
I20260927 22:41:02.230856 6893 leader_election.cc:302] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f6813ff604460b885b47e5188e2ba3; no voters:
I20260927 22:41:02.231446 6893 leader_election.cc:288] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:02.231647 6898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:02.232296 6898 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [term 1 LEADER]: Becoming Leader. State: Replica: b9f6813ff604460b885b47e5188e2ba3, State: Running, Role: LEADER
I20260927 22:41:02.232883 6898 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6813ff604460b885b47e5188e2ba3" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46331 } }
I20260927 22:41:02.233791 6893 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:02.236179 6899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f6813ff604460b885b47e5188e2ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6813ff604460b885b47e5188e2ba3" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46331 } } }
I20260927 22:41:02.236148 6900 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f6813ff604460b885b47e5188e2ba3. Latest consensus state: current_term: 1 leader_uuid: "b9f6813ff604460b885b47e5188e2ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6813ff604460b885b47e5188e2ba3" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46331 } } }
I20260927 22:41:02.236797 6899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:02.236835 6900 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:02.238945 6906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:02.244299 6906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:02.253137 6906 catalog_manager.cc:1347] Generated new cluster ID: 665475cdf9b8429f8a8fc9776c8a0ac2
I20260927 22:41:02.253286 6906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:02.278417 6906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:02.279804 6906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:02.290056 6906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f6813ff604460b885b47e5188e2ba3: Generated new TSK 0
I20260927 22:41:02.291159 6906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:41:02.348369 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46331
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:02.684976 6919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:02.685310 6919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:02.685474 6919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:02.692919 6919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:02.693074 6919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:02.693254 6919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:02.703289 6919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:02.704687 6919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:02.706022 6919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:02.719300 6924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:02.719336 6925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:02.719333 6927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:02.725929 6919 server_base.cc:1034] running on GCE node
I20260927 22:41:02.726645 6919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:02.727738 6919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:02.729007 6919 hybrid_clock.cc:648] HybridClock initialized: now 1790548862728908 us; error 67 us; skew 500 ppm
I20260927 22:41:02.729336 6919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:02 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:02.732576 6919 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:41:02.734200 6919 webserver.cc:460] Webserver started at http://127.1.53.65:33073/ using document root <none> and password file <none>
I20260927 22:41:02.735059 6919 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:02.735203 6919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:02.735510 6919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:02.738540 6919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "b8db84ba3a86449cab49c1917dc75598"
format_stamp: "Formatted at 2026-09-27 22:41:02 on dist-test-slave-pp80"
server_key: "f8a2b90d4c0d4222034c568491151126"
server_key_iv: "303f2a90e362422291fd2ca479918ce0"
server_key_version: "encryptionkey@0"
I20260927 22:41:02.739415 6919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "b8db84ba3a86449cab49c1917dc75598"
format_stamp: "Formatted at 2026-09-27 22:41:02 on dist-test-slave-pp80"
server_key: "f8a2b90d4c0d4222034c568491151126"
server_key_iv: "303f2a90e362422291fd2ca479918ce0"
server_key_version: "encryptionkey@0"
I20260927 22:41:02.744602 6919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:41:02.748075 6934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:02.749495 6919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:02.749675 6919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "b8db84ba3a86449cab49c1917dc75598"
format_stamp: "Formatted at 2026-09-27 22:41:02 on dist-test-slave-pp80"
server_key: "f8a2b90d4c0d4222034c568491151126"
server_key_iv: "303f2a90e362422291fd2ca479918ce0"
server_key_version: "encryptionkey@0"
I20260927 22:41:02.749881 6919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:02.769117 6919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:02.773552 6919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:02.773871 6919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:02.774823 6919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:02.776227 6919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:02.776355 6919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:02.776512 6919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:02.776583 6919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:02.810824 6919 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:44305
I20260927 22:41:02.810881 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:44305 every 8 connection(s)
I20260927 22:41:02.812098 6919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:02.820611 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 6919
I20260927 22:41:02.820837 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:02.821658 1237 external_mini_cluster.cc:1428] Setting key d28893276627680829667caebb3f3b0c
Sep 27 22:41:02 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548862, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:02.834029 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:02.814852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:47251 (local address 127.1.53.126:46331)
0927 22:41:02.815340 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:02.815346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:02.816453 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 22:41:02.818210 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:02.818214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:02.818293 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:02.818396 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:02.819929 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:02.820721 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:02.823016 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:02.823828 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:02.823965 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:02.828309 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:02.828315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:02.828319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:02.828351 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:02.830886 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:02.831624 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:02.831630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:02.831641 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:02.831752 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:02.832185 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:02.832191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:02.832194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:02.832531 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:02.832684 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:02.833413 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 22:41:02.833636 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260927 22:41:02.835253 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:02.815467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46331 (local address 127.1.53.65:47251)
0927 22:41:02.816226 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:02.816341 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:02.817946 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:02.818620 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:02.818628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:02.819224 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:02.819763 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:02.819778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:02.820997 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:02.821004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:02.822863 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:02.822874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:02.824132 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:02.824139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:02.824470 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:02.825257 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:02.825299 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:02.828030 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:02.831043 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:02.831052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:02.831069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:02.831491 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:02.831860 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:02.831866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:02.831870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:02.832072 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:02.832657 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:02.832665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:02.833037 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 22:41:02.833313 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 22:41:02.833972 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":289,"threads_started":1}
I20260927 22:41:02.836835 7047 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46331
I20260927 22:41:02.837177 7047 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:02.838146 7047 heartbeater.cc:511] Master 127.1.53.126:46331 requested a full tablet report, sending...
I20260927 22:41:02.840543 6858 ts_manager.cc:194] Registered new tserver with Master: b8db84ba3a86449cab49c1917dc75598 (127.1.53.65:44305)
I20260927 22:41:02.842012 6858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:47251
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:41:02.871778 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46331
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:03.199456 7053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:03.199774 7053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:03.199976 7053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:03.207497 7053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:03.207638 7053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:03.207824 7053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:03.217693 7053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:03.218940 7053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:03.220335 7053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:03.233047 7058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:03.233167 7061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:03.233049 7059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:03.233909 7053 server_base.cc:1034] running on GCE node
I20260927 22:41:03.234503 7053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:03.235625 7053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:03.236860 7053 hybrid_clock.cc:648] HybridClock initialized: now 1790548863236817 us; error 43 us; skew 500 ppm
I20260927 22:41:03.237229 7053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.240703 7053 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:41:03.242290 7053 webserver.cc:460] Webserver started at http://127.1.53.66:33183/ using document root <none> and password file <none>
I20260927 22:41:03.243000 7053 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:03.243141 7053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:03.243450 7053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:03.246553 7053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "ad94e2d5b26649e786ee8ff6ec3d3dd3"
format_stamp: "Formatted at 2026-09-27 22:41:03 on dist-test-slave-pp80"
server_key: "546eac35b4da2c308a16dcd2d5410388"
server_key_iv: "62927337e27249273b8c733ecd69717f"
server_key_version: "encryptionkey@0"
I20260927 22:41:03.247336 7053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "ad94e2d5b26649e786ee8ff6ec3d3dd3"
format_stamp: "Formatted at 2026-09-27 22:41:03 on dist-test-slave-pp80"
server_key: "546eac35b4da2c308a16dcd2d5410388"
server_key_iv: "62927337e27249273b8c733ecd69717f"
server_key_version: "encryptionkey@0"
I20260927 22:41:03.252476 7053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:41:03.256142 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:03.257659 7053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:41:03.257848 7053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "ad94e2d5b26649e786ee8ff6ec3d3dd3"
format_stamp: "Formatted at 2026-09-27 22:41:03 on dist-test-slave-pp80"
server_key: "546eac35b4da2c308a16dcd2d5410388"
server_key_iv: "62927337e27249273b8c733ecd69717f"
server_key_version: "encryptionkey@0"
I20260927 22:41:03.258064 7053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:03.287148 7053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:03.291749 7053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:03.292079 7053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:03.293151 7053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:03.294597 7053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:03.294713 7053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:03.294878 7053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:03.295038 7053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:03.329069 7053 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:38413
I20260927 22:41:03.329129 7180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:38413 every 8 connection(s)
I20260927 22:41:03.330338 7053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:03.339902 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7053
I20260927 22:41:03.340070 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:03.340575 1237 external_mini_cluster.cc:1428] Setting key 7e44861f9ef0061aa03cf6f8ff6b29a2
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:03.353271 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.333136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:55015 (local address 127.1.53.126:46331)
0927 22:41:03.333306 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.333310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.334679 (+ 1369us) server_negotiation.cc:407] Connection header received
0927 22:41:03.336804 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.336809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.336879 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.336990 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.338585 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.339402 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.341484 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.342070 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.342172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.346282 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:03.346291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:03.346299 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:03.346344 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:03.349601 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.350650 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.350655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.350657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.350784 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.351338 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.351342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.351344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.351620 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:03.351733 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:03.352767 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0927 22:41:03.353012 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:41:03.354871 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.333554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46331 (local address 127.1.53.66:55015)
0927 22:41:03.334506 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:03.334554 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:03.336535 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:03.337196 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:03.337205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:03.337800 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:03.338421 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:03.338440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.339544 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.339548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:03.341334 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:03.341350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.342314 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.342322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:03.342656 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.343442 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:03.343466 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:03.346026 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:03.349773 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:03.349786 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:03.349808 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:03.350501 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:03.350911 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:03.350919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:03.350927 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:03.351186 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:03.351752 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:03.351764 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:03.352288 (+ 524us) client_negotiation.cc:769] Sending connection context
0927 22:41:03.352622 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 22:41:03.353256 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":323,"threads_started":1}
I20260927 22:41:03.356493 7181 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46331
I20260927 22:41:03.356824 7181 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:03.357591 7181 heartbeater.cc:511] Master 127.1.53.126:46331 requested a full tablet report, sending...
I20260927 22:41:03.359226 6858 ts_manager.cc:194] Registered new tserver with Master: ad94e2d5b26649e786ee8ff6ec3d3dd3 (127.1.53.66:38413)
I20260927 22:41:03.360057 6858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:55015
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:41:03.391873 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46331
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:03.732450 7186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:03.732736 7186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:03.732887 7186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:03.740545 7186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:03.740658 7186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:03.740820 7186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:03.750972 7186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:03.752385 7186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:03.753870 7186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:03.767843 7191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:03.767792 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:03.767820 7194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:03.768597 7186 server_base.cc:1034] running on GCE node
I20260927 22:41:03.769248 7186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:03.770265 7186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:03.771497 7186 hybrid_clock.cc:648] HybridClock initialized: now 1790548863771439 us; error 55 us; skew 500 ppm
I20260927 22:41:03.771862 7186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.775075 7186 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:41:03.776628 7186 webserver.cc:460] Webserver started at http://127.1.53.67:46207/ using document root <none> and password file <none>
I20260927 22:41:03.777323 7186 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:03.777467 7186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:03.777781 7186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:03.780764 7186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "ab42125406064d27afae6c6cf9b01627"
format_stamp: "Formatted at 2026-09-27 22:41:03 on dist-test-slave-pp80"
server_key: "9409c319dda952702c935d523a6c6642"
server_key_iv: "bf263780811eff7efdf85d9783ce45d5"
server_key_version: "encryptionkey@0"
I20260927 22:41:03.781554 7186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "ab42125406064d27afae6c6cf9b01627"
format_stamp: "Formatted at 2026-09-27 22:41:03 on dist-test-slave-pp80"
server_key: "9409c319dda952702c935d523a6c6642"
server_key_iv: "bf263780811eff7efdf85d9783ce45d5"
server_key_version: "encryptionkey@0"
I20260927 22:41:03.787303 7186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:41:03.790925 7201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:03.792528 7186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:03.792707 7186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "ab42125406064d27afae6c6cf9b01627"
format_stamp: "Formatted at 2026-09-27 22:41:03 on dist-test-slave-pp80"
server_key: "9409c319dda952702c935d523a6c6642"
server_key_iv: "bf263780811eff7efdf85d9783ce45d5"
server_key_version: "encryptionkey@0"
I20260927 22:41:03.792920 7186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:03.810091 7186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:03.814492 7186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:03.814776 7186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:03.815698 7186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:03.817041 7186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:03.817142 7186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:03.817302 7186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:03.817373 7186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:03.845197 7047 heartbeater.cc:503] Master 127.1.53.126:46331 was elected leader, sending a full tablet report...
I20260927 22:41:03.850982 7186 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39577
I20260927 22:41:03.851068 7313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39577 every 8 connection(s)
I20260927 22:41:03.852222 7186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:03.853434 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7186
I20260927 22:41:03.853624 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:03.854269 1237 external_mini_cluster.cc:1428] Setting key be23e933f783785a06b9777810464c68
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:03.873682 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.855544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:50509 (local address 127.1.53.126:46331)
0927 22:41:03.856299 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.856306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.857375 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 22:41:03.859335 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.859339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.859406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.859501 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.861168 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.862065 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.863690 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.864068 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.864164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.867813 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:03.867822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:03.867827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:03.867865 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:03.870680 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.871412 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.871424 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.871428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.871546 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.871982 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.871991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.871993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.872308 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:03.872454 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:03.873115 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 22:41:03.873315 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":515,"threads_started":1}
I20260927 22:41:03.874644 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.856274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46331 (local address 127.1.53.67:50509)
0927 22:41:03.857223 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:03.857271 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:03.859090 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:03.859714 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:03.859722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:03.860373 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:03.861008 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:03.861026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.862218 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.862222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:03.863566 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:03.863578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.864368 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.864374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:03.864580 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.865300 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:03.865321 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:03.867499 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:03.870825 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:03.870834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:03.870852 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:03.871285 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:03.871653 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:03.871659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:03.871663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:03.871851 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:03.872428 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:03.872443 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:03.872763 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:41:03.873020 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 22:41:03.873540 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":405,"threads_started":1}
I20260927 22:41:03.875984 7314 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46331
I20260927 22:41:03.876407 7314 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:03.877192 7314 heartbeater.cc:511] Master 127.1.53.126:46331 requested a full tablet report, sending...
I20260927 22:41:03.878654 6858 ts_manager.cc:194] Registered new tserver with Master: ab42125406064d27afae6c6cf9b01627 (127.1.53.67:39577)
I20260927 22:41:03.879530 6858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:50509
I20260927 22:41:03.882385 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/test-admin.keytab.
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.910471 1237 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:41:03.916092 7322 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:03.913494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.1.53.65:44305)
0927 22:41:03.914205 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.914212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.914242 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:41:03.914301 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.914307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.914523 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.914653 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.915692 (+ 1039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52852: BlockingRecv error: recv got EOF from 127.0.0.1:52852 (error 108)
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":372,"threads_started":1}
W20260927 22:41:03.916136 1237 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.917974 1237 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.919410 1237 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:41:03.935727 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.921144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.1.53.65:44305)
0927 22:41:03.921377 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.921383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.921585 (+ 202us) server_negotiation.cc:407] Connection header received
0927 22:41:03.921788 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.921792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.921850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.921970 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.922916 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.923856 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.925144 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.925730 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.925894 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.929694 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:03.929706 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:03.929721 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:03.929754 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:03.932654 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.933425 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.933432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.933435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.933554 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.933970 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.933978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.933982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.934469 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:03.934814 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:03.935154 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 22:41:03.935423 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:41:03.941844 7322 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:03.940127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.1.53.65:44305)
0927 22:41:03.940346 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.940352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.940488 (+ 136us) server_negotiation.cc:407] Connection header received
0927 22:41:03.940688 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.940692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.940745 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.940843 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.941542 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52878: BlockingRecv error: recv got EOF from 127.0.0.1:52878 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 22:41:03.942058 1237 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.943775 1237 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.945112 1237 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:41:03.959214 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.946447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.1.53.65:44305)
0927 22:41:03.946629 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.946637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.946982 (+ 345us) server_negotiation.cc:407] Connection header received
0927 22:41:03.947175 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.947179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.947231 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.947326 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.948258 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.948968 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.950182 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.950589 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.950719 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.954322 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:03.954326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:03.954329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:03.954370 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:03.956820 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.957420 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.957424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.957427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.957542 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.957936 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.957941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.957943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.958220 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:03.958444 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:03.958662 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 22:41:03.958920 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 22:41:03.964007 7322 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:03.962188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.1.53.65:44305)
0927 22:41:03.962394 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.962404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.962695 (+ 291us) server_negotiation.cc:407] Connection header received
0927 22:41:03.962899 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.962904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.962982 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.963085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.963787 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52888: BlockingRecv error: recv got EOF from 127.0.0.1:52888 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:41:03.964231 1237 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790548812741971-1237-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.965785 1237 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:03.967394 1237 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:41:03 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:41:03.981273 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.968664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.1.53.65:44305)
0927 22:41:03.968876 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.968881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.969123 (+ 242us) server_negotiation.cc:407] Connection header received
0927 22:41:03.969298 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.969310 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.969361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.969455 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.970370 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.971064 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.972287 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.972698 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.972824 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.976355 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:03.976360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:03.976363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:03.976391 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:03.978977 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.979576 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.979580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.979582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.979724 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.980034 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.980037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.980039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.980370 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:03.980593 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:03.980764 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:41:03.980964 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:41:03.982754 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6919
I20260927 22:41:03.994442 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:03.983410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.1.53.65:44305)
0927 22:41:03.983548 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:03.983552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:03.983887 (+ 335us) server_negotiation.cc:407] Connection header received
0927 22:41:03.984011 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:03.984014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:03.984108 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:03.984183 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:03.985115 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.985987 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.987209 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:03.987625 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:03.987715 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:03.989612 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:03.989616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:03.989626 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:03.989653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:03.992134 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.992852 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.992856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.992858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.993010 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:03.993349 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:03.993352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:03.993354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:03.993606 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:03.993796 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:03.994040 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 22:41:03.994195 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:41:03.995043 7042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:04.079988 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6919
I20260927 22:41:04.096431 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7053
Sep 27 22:41:04 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:41:04.111908 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:04.097379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.1.53.66:38413)
0927 22:41:04.097891 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:04.097910 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:04.097941 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:41:04.098014 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:04.098020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:04.098234 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:04.098415 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:04.099345 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:04.100403 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:04.101607 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:04.102149 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:04.102288 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:04.106330 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:04.106341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:04.106356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:04.106389 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:04.109174 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:04.109829 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:04.109838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:04.109840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:04.109951 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:04.110350 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:04.110356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:04.110359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:04.110694 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:04.110995 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:04.111334 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:41:04.111588 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":265,"threads_started":1}
I20260927 22:41:04.113376 7176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:04.197230 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7053
I20260927 22:41:04.212956 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7186
Sep 27 22:41:04 dist-test-slave-pp80 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:41:04.229346 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:04.214033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.1.53.67:39577)
0927 22:41:04.214621 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:04.214628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:04.214657 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:41:04.214718 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:04.214724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:04.214934 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:04.215104 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:04.216156 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:04.217251 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:04.218489 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:04.219065 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:04.219163 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:04.223420 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:04.223432 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:04.223447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:04.223479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:04.226573 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:04.227313 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:04.227321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:04.227323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:04.227445 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:04.227849 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:04.227852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:04.227854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:04.228217 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:04.228577 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:04.228818 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:41:04.229056 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260927 22:41:04.230700 7309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:04.316563 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7186
I20260927 22:41:04.332434 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 6826
I20260927 22:41:04.333725 6888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:04.435523 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 6826
2026-09-27T22:41:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3975 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:04 dist-test-slave-pp80 krb5kdc[7342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:04 dist-test-slave-pp80 krb5kdc[7342](info): set up 2 sockets
Sep 27 22:41:04 dist-test-slave-pp80 krb5kdc[7342](info): commencing operation
krb5kdc: starting...
W20260927 22:41:05.517109 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:05 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab.
Entry for principal oryx/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab.
I20260927 22:41:05.731796 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42075
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab
--principal=oryx/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:06.055040 7355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:06.055414 7355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:06.055569 7355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:06.062902 7355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:06.063066 7355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:06.063169 7355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:06.063272 7355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:06.063349 7355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:06.073274 7355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab
--principal=oryx/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:06.074663 7355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:06.076005 7355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:06.087939 7360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:06.087946 7361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:06.088505 7363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:06.089192 7355 server_base.cc:1034] running on GCE node
I20260927 22:41:06.090008 7355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:06.091450 7355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:06.092703 7355 hybrid_clock.cc:648] HybridClock initialized: now 1790548866092660 us; error 59 us; skew 500 ppm
I20260927 22:41:06.093111 7355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:06 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548866, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:06.100750 7355 init.cc:377] Logged in from keytab as oryx/127.1.53.126@KRBTEST.COM (short username oryx)
I20260927 22:41:06.102442 7355 webserver.cc:460] Webserver started at http://127.1.53.126:38741/ using document root <none> and password file <none>
I20260927 22:41:06.103176 7355 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:06.103328 7355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:06.103643 7355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:06.106734 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "057aa11ad898437fa599b88a33ccb839"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "bceb51c034d4526cdb9e1d048b61c627"
server_key_iv: "37ab1a42ab925da69d7c58523e4c31ea"
server_key_version: "encryptionkey@0"
I20260927 22:41:06.107548 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "057aa11ad898437fa599b88a33ccb839"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "bceb51c034d4526cdb9e1d048b61c627"
server_key_iv: "37ab1a42ab925da69d7c58523e4c31ea"
server_key_version: "encryptionkey@0"
I20260927 22:41:06.112972 7355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:41:06.116782 7370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:06.118235 7355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:41:06.118431 7355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "057aa11ad898437fa599b88a33ccb839"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "bceb51c034d4526cdb9e1d048b61c627"
server_key_iv: "37ab1a42ab925da69d7c58523e4c31ea"
server_key_version: "encryptionkey@0"
I20260927 22:41:06.118638 7355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:06.156065 7355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:06.160748 7355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:06.161074 7355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:06.181450 7355 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42075
I20260927 22:41:06.181479 7421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42075 every 8 connection(s)
I20260927 22:41:06.182968 7355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:06.186879 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:06.192703 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7355
I20260927 22:41:06.192903 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:06.193614 1237 external_mini_cluster.cc:1428] Setting key 96c17bea1efe7846f1b4372ea14bec0d
I20260927 22:41:06.198138 7422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Bootstrap starting.
I20260927 22:41:06.201946 7422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:06.203222 7422 log.cc:824] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:06.206897 7422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: No bootstrap required, opened a new log
Sep 27 22:41:06 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:06.215021 7422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } }
I20260927 22:41:06.215433 7422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:06.215572 7422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057aa11ad898437fa599b88a33ccb839, State: Initialized, Role: FOLLOWER
I20260927 22:41:06.216316 7422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } }
I20260927 22:41:06.216588 7422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:06.216724 7422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:06.216890 7422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:06.217815 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:06.195707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.1.53.126:42075)
0927 22:41:06.196535 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:06.196547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:06.196588 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:41:06.198229 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:06.198252 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:06.198695 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:06.199020 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:06.200882 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:06.202305 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:06.204131 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:06.205054 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:06.205190 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:06.210072 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:06.210080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:06.210098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:06.210141 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:06.213378 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:06.214089 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:06.214096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:06.214102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:06.214261 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:06.214644 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:06.214651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:06.214654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:06.215173 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:06.215495 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:06.215893 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 22:41:06.216375 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":392,"threads_started":1}
I20260927 22:41:06.219570 7422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } }
I20260927 22:41:06.220003 7422 leader_election.cc:302] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 057aa11ad898437fa599b88a33ccb839; no voters:
I20260927 22:41:06.220660 7422 leader_election.cc:288] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:06.220803 7427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:06.221489 7427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 LEADER]: Becoming Leader. State: Replica: 057aa11ad898437fa599b88a33ccb839, State: Running, Role: LEADER
I20260927 22:41:06.222066 7427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } }
I20260927 22:41:06.222772 7422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:06.225291 7428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } }
I20260927 22:41:06.225319 7429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057aa11ad898437fa599b88a33ccb839. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } }
I20260927 22:41:06.225730 7429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:06.225731 7428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:06.230760 7436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:06.235713 7436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:06.244796 7436 catalog_manager.cc:1347] Generated new cluster ID: cadfcac557c94fdeb02e19f5913049bd
I20260927 22:41:06.244918 7436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:06.264420 7436 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:06.266283 7436 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:06.274356 7436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Generated new TSK 0
I20260927 22:41:06.275342 7436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab.
Entry for principal oryx/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab.
I20260927 22:41:06.330464 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42075
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:06.659272 7448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:06.659559 7448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:06.659713 7448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:06.666981 7448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:06.667097 7448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:06.667284 7448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:06.677155 7448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:06.678496 7448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:06.679841 7448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:06.693240 7456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:06.693404 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:06.693662 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:06.694479 7448 server_base.cc:1034] running on GCE node
I20260927 22:41:06.695099 7448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:06.696123 7448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:06.697377 7448 hybrid_clock.cc:648] HybridClock initialized: now 1790548866697312 us; error 58 us; skew 500 ppm
I20260927 22:41:06.697748 7448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:06 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548866, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:06.701072 7448 init.cc:377] Logged in from keytab as oryx/127.1.53.65@KRBTEST.COM (short username oryx)
I20260927 22:41:06.702672 7448 webserver.cc:460] Webserver started at http://127.1.53.65:42837/ using document root <none> and password file <none>
I20260927 22:41:06.703413 7448 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:06.703557 7448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:06.703884 7448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:06.707085 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "ded01630111646d1a7fa7d6cc9c12b7a"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "5d279f7a74ba5091d21659691838d5d5"
server_key_iv: "5883e0c6fff80efa994624f51fff4786"
server_key_version: "encryptionkey@0"
I20260927 22:41:06.707866 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "ded01630111646d1a7fa7d6cc9c12b7a"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "5d279f7a74ba5091d21659691838d5d5"
server_key_iv: "5883e0c6fff80efa994624f51fff4786"
server_key_version: "encryptionkey@0"
I20260927 22:41:06.713310 7448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:41:06.716830 7463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:06.718429 7448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:41:06.718647 7448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "ded01630111646d1a7fa7d6cc9c12b7a"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "5d279f7a74ba5091d21659691838d5d5"
server_key_iv: "5883e0c6fff80efa994624f51fff4786"
server_key_version: "encryptionkey@0"
I20260927 22:41:06.718986 7448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:06.754326 7448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:06.758822 7448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:06.759135 7448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:06.760140 7448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:06.761519 7448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:06.761595 7448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:06.761758 7448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:06.761842 7448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:06.796023 7448 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:44621
I20260927 22:41:06.796141 7575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:44621 every 8 connection(s)
I20260927 22:41:06.797333 7448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:06.803552 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7448
I20260927 22:41:06.803772 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:06.804447 1237 external_mini_cluster.cc:1428] Setting key 770db5505e907abbf83c73433212ffff
Sep 27 22:41:06 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548866, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:06.818894 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:06.799758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:58055 (local address 127.1.53.126:42075)
0927 22:41:06.800188 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:06.800192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:06.801216 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 22:41:06.803453 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:06.803459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:06.803586 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:06.803711 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:06.805394 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:06.806310 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:06.807908 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:06.808666 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:06.808810 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:06.812700 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:06.812708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:06.812713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:06.812758 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:06.815572 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:06.816574 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:06.816578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:06.816580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:06.816695 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:06.817201 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:06.817204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:06.817206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:06.817449 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:06.817555 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:06.818469 (+ 914us) server_negotiation.cc:299] Negotiation successful
0927 22:41:06.818632 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260927 22:41:06.820518 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:06.800408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:58055)
0927 22:41:06.801055 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:06.801096 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:06.803073 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:06.804012 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:06.804020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:06.804664 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:06.805246 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:06.805263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:06.806478 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:06.806482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:06.807748 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:06.807759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:06.808915 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:06.808920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:06.809118 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:06.809859 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:06.809887 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:06.812458 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:06.815724 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:06.815738 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:06.815758 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:06.816447 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:06.816804 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:06.816811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:06.816818 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:06.817085 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:06.817574 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:06.817585 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:06.818055 (+ 470us) client_negotiation.cc:769] Sending connection context
0927 22:41:06.818355 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 22:41:06.818969 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
I20260927 22:41:06.822189 7576 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42075
I20260927 22:41:06.822587 7576 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:06.823521 7576 heartbeater.cc:511] Master 127.1.53.126:42075 requested a full tablet report, sending...
I20260927 22:41:06.825737 7387 ts_manager.cc:194] Registered new tserver with Master: ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65:44621)
WARNING: no policy specified for oryx/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:41:06.827298 7387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:58055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab.
Entry for principal oryx/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab.
I20260927 22:41:06.856148 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42075
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:07.188922 7582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:07.189275 7582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:07.189453 7582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:07.197219 7582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:07.197389 7582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:07.197575 7582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:07.208057 7582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:07.209561 7582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:07.210959 7582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:07.224746 7587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:07.224773 7590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:07.224958 7588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:07.226527 7582 server_base.cc:1034] running on GCE node
I20260927 22:41:07.227231 7582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:07.228356 7582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:07.229585 7582 hybrid_clock.cc:648] HybridClock initialized: now 1790548867229525 us; error 63 us; skew 500 ppm
I20260927 22:41:07.230032 7582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:07 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548867, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:07.233413 7582 init.cc:377] Logged in from keytab as oryx/127.1.53.66@KRBTEST.COM (short username oryx)
I20260927 22:41:07.234949 7582 webserver.cc:460] Webserver started at http://127.1.53.66:38563/ using document root <none> and password file <none>
I20260927 22:41:07.235673 7582 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:07.235817 7582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:07.236121 7582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:07.239091 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "77fc447e03a044c0b06dad1b4ed79a75"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "ce7dd5334242b3668382ca43165cb565"
server_key_iv: "801ecc69cc93a7b2b47dc229d60ee9f3"
server_key_version: "encryptionkey@0"
I20260927 22:41:07.239887 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "77fc447e03a044c0b06dad1b4ed79a75"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "ce7dd5334242b3668382ca43165cb565"
server_key_iv: "801ecc69cc93a7b2b47dc229d60ee9f3"
server_key_version: "encryptionkey@0"
I20260927 22:41:07.245226 7582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:41:07.248963 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:07.250351 7582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:41:07.250547 7582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "77fc447e03a044c0b06dad1b4ed79a75"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "ce7dd5334242b3668382ca43165cb565"
server_key_iv: "801ecc69cc93a7b2b47dc229d60ee9f3"
server_key_version: "encryptionkey@0"
I20260927 22:41:07.250779 7582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:07.276885 7582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:07.281543 7582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:07.281852 7582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:07.282868 7582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:07.284323 7582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:07.284401 7582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:07.284518 7582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:07.284575 7582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:07.317973 7582 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:41405
I20260927 22:41:07.318022 7709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:41405 every 8 connection(s)
I20260927 22:41:07.319237 7582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:07.327716 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7582
I20260927 22:41:07.327883 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:07.328500 1237 external_mini_cluster.cc:1428] Setting key e457ff196868994ca9a8e0693c769f4f
Sep 27 22:41:07 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548867, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:07.340759 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:07.321742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:43131 (local address 127.1.53.126:42075)
0927 22:41:07.322132 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:07.322137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:07.323331 (+ 1194us) server_negotiation.cc:407] Connection header received
0927 22:41:07.325436 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:07.325442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:07.325532 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:07.325649 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:07.327127 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.327885 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.329990 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.330625 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.330741 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:07.334907 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:07.334912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:07.334915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:07.334945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:07.337603 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:07.338340 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:07.338352 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:07.338354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:07.338492 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:07.338975 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:07.338984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:07.338986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:07.339252 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:07.339417 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:07.340135 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 22:41:07.340399 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":192,"threads_started":1}
I20260927 22:41:07.341816 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:07.322270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:43131)
0927 22:41:07.323157 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:07.323217 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:07.325157 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:07.325905 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:07.325914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:07.326417 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:07.326959 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:07.326976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.328088 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.328093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:07.329831 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:07.329847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.330867 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.330874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:07.331165 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:07.331896 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:07.331943 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:07.334679 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:07.337749 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:07.337758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:07.337776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:07.338210 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:07.338615 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:07.338620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:07.338623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:07.338813 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:07.339387 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:07.339396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:07.339757 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 22:41:07.340034 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 22:41:07.340615 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":355,"threads_started":1}
I20260927 22:41:07.343585 7710 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42075
I20260927 22:41:07.343968 7710 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:07.344851 7710 heartbeater.cc:511] Master 127.1.53.126:42075 requested a full tablet report, sending...
I20260927 22:41:07.346691 7387 ts_manager.cc:194] Registered new tserver with Master: 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:07.347954 7387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:43131
WARNING: no policy specified for oryx/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab.
Entry for principal oryx/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab.
I20260927 22:41:07.375926 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42075
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:07.701552 7716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:07.701889 7716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:07.702061 7716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:07.709461 7716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:07.709622 7716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:07.709815 7716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:07.719664 7716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:07.721210 7716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:07.722674 7716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:07.736119 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:07.736078 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:07.736150 7724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:07.736979 7716 server_base.cc:1034] running on GCE node
I20260927 22:41:07.737597 7716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:07.738550 7716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:07.739786 7716 hybrid_clock.cc:648] HybridClock initialized: now 1790548867739704 us; error 80 us; skew 500 ppm
I20260927 22:41:07.740137 7716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:07 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548867, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:07.743412 7716 init.cc:377] Logged in from keytab as oryx/127.1.53.67@KRBTEST.COM (short username oryx)
I20260927 22:41:07.744980 7716 webserver.cc:460] Webserver started at http://127.1.53.67:39307/ using document root <none> and password file <none>
I20260927 22:41:07.745671 7716 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:07.745826 7716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:07.746140 7716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:07.749079 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "3eda5af2fa284054aaf632a0ffb4c870"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "577163846cce90a834978ff3b19fe0a3"
server_key_iv: "89ce0a5f5f5d4aea3fbb1f261c498612"
server_key_version: "encryptionkey@0"
I20260927 22:41:07.749818 7716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "3eda5af2fa284054aaf632a0ffb4c870"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "577163846cce90a834978ff3b19fe0a3"
server_key_iv: "89ce0a5f5f5d4aea3fbb1f261c498612"
server_key_version: "encryptionkey@0"
I20260927 22:41:07.756038 7716 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 22:41:07.759574 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:07.761023 7716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:07.761181 7716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "3eda5af2fa284054aaf632a0ffb4c870"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "577163846cce90a834978ff3b19fe0a3"
server_key_iv: "89ce0a5f5f5d4aea3fbb1f261c498612"
server_key_version: "encryptionkey@0"
I20260927 22:41:07.761404 7716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:07.775803 7716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:07.780176 7716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:07.780539 7716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:07.781531 7716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:07.782814 7716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:07.782883 7716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:07.783027 7716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:07.783090 7716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:07.816075 7716 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:32915
I20260927 22:41:07.816361 7843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:32915 every 8 connection(s)
I20260927 22:41:07.817435 7716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:07.825747 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7716
I20260927 22:41:07.825950 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:07.826421 1237 external_mini_cluster.cc:1428] Setting key 7d5b49ae46e4ba821ebda5d99bb5ca89
I20260927 22:41:07.830782 7576 heartbeater.cc:503] Master 127.1.53.126:42075 was elected leader, sending a full tablet report...
Sep 27 22:41:07 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548867, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:07.838195 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:07.820308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:43389 (local address 127.1.53.126:42075)
0927 22:41:07.820493 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:07.820498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:07.821691 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 22:41:07.823857 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:07.823863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:07.823998 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:07.824114 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:07.825625 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.826486 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.828456 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.828929 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.829065 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:07.832952 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:07.832956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:07.832959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:07.832990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:07.835390 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:07.836135 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:07.836139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:07.836142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:07.836346 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:07.836740 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:07.836743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:07.836745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:07.837018 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:07.837131 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:07.837794 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 22:41:07.837969 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:41:07.839201 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:07.820753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:43389)
0927 22:41:07.821525 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:07.821568 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:07.823544 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:07.824381 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:07.824390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:07.824890 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:07.825461 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:07.825478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.826676 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.826682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:07.828305 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:07.828317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:07.829151 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:07.829157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:07.829457 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:07.830181 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:07.830203 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:07.832744 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:07.835521 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:07.835527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:07.835542 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:07.836000 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:07.836451 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:07.836454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:07.836456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:07.836640 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:07.837122 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:07.837127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:07.837440 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 22:41:07.837666 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 22:41:07.838047 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":292,"threads_started":1}
I20260927 22:41:07.840794 7844 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42075
I20260927 22:41:07.841146 7844 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:07.841845 7844 heartbeater.cc:511] Master 127.1.53.126:42075 requested a full tablet report, sending...
I20260927 22:41:07.843245 7387 ts_manager.cc:194] Registered new tserver with Master: 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915)
I20260927 22:41:07.844024 7387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:43389
I20260927 22:41:07.854795 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.125.keytab.
Entry for principal oryx/127.1.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.125.keytab.
I20260927 22:41:07.903472 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:37173
--webserver_interface=127.1.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.125.keytab
--principal=oryx/127.1.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42075,127.1.53.125:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:08.224977 7849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:08.225267 7849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:08.225467 7849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:08.232899 7849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:08.233000 7849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:08.233079 7849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:08.233175 7849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:08.233273 7849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:08.243153 7849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42075,127.1.53.125:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.125.keytab
--principal=oryx/127.1.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
W20260927 22:41:08.243535 7849 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.53.126:42075,127.1.53.125:37173'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:41:08.244701 7849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:08.246064 7849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:08.261219 7857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:08.262776 7854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:08.265209 7855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:08.266597 7849 server_base.cc:1034] running on GCE node
I20260927 22:41:08.267765 7849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:08.269976 7849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:08.271271 7849 hybrid_clock.cc:648] HybridClock initialized: now 1790548868271174 us; error 64 us; skew 500 ppm
I20260927 22:41:08.271718 7849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:08.276038 7849 init.cc:377] Logged in from keytab as oryx/127.1.53.125@KRBTEST.COM (short username oryx)
I20260927 22:41:08.277761 7849 webserver.cc:460] Webserver started at http://127.1.53.125:40245/ using document root <none> and password file <none>
I20260927 22:41:08.278798 7849 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:08.278934 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:08.279321 7849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:08.283427 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/instance:
uuid: "2f90e4b26e334500ba8d3b211087d65c"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "1dc2c97d36d8c723ad2108238ae09b85"
server_key_iv: "7f972afab11e1748a06a64a4bad47b21"
server_key_version: "encryptionkey@0"
I20260927 22:41:08.284212 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal/instance:
uuid: "2f90e4b26e334500ba8d3b211087d65c"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "1dc2c97d36d8c723ad2108238ae09b85"
server_key_iv: "7f972afab11e1748a06a64a4bad47b21"
server_key_version: "encryptionkey@0"
I20260927 22:41:08.289429 7849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:41:08.292804 7864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:08.294127 7849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:41:08.294301 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
uuid: "2f90e4b26e334500ba8d3b211087d65c"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "1dc2c97d36d8c723ad2108238ae09b85"
server_key_iv: "7f972afab11e1748a06a64a4bad47b21"
server_key_version: "encryptionkey@0"
I20260927 22:41:08.294498 7849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:08.309494 7849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:08.313989 7849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:08.314257 7849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:08.333510 7849 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.125:37173
I20260927 22:41:08.333535 7915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.125:37173 every 8 connection(s)
I20260927 22:41:08.334844 7849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
I20260927 22:41:08.338656 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:08.343829 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7849
I20260927 22:41:08.344918 7916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:41:08.350756 7710 heartbeater.cc:503] Master 127.1.53.126:42075 was elected leader, sending a full tablet report...
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.125@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
I20260927 22:41:08.365198 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.348072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.1.53.126:42075)
0927 22:41:08.348569 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.348576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.349176 (+ 600us) server_negotiation.cc:407] Connection header received
0927 22:41:08.349575 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.349584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.349676 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.349782 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.351547 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.352738 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.354411 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.354875 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.354968 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.358647 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.358654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.358659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.358698 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.361667 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.362596 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.362600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.362602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.362709 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.363179 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.363183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.363186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.363615 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.363769 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.364629 (+ 860us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.364841 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260927 22:41:08.366576 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.348119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48994)
0927 22:41:08.349022 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.349060 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.349355 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.349867 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.349874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.350764 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:08.351381 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.351409 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.352863 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.352868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.354275 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.354287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.355088 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.355097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.355259 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.356033 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:08.356054 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:08.358451 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:08.361796 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.361805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.361818 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.362482 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.362817 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.362824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.362828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.363069 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.363768 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:08.363776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:08.364204 (+ 428us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.364539 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.365035 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":8,"thread_start_us":480,"threads_started":1}
I20260927 22:41:08.366593 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.346024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.1.53.125:37173)
0927 22:41:08.346813 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.346829 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.346878 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:41:08.349086 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.349108 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.349468 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.349693 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.351604 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.353092 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.354477 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.355047 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.355212 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.359725 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.359734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.359747 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.359774 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.363051 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.363762 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.363770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.363773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.363909 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.364318 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.364324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.364327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.364723 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.365027 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.365412 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.365651 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":581,"spinlock_wait_cycles":28288,"thread_start_us":350,"threads_started":1}
I20260927 22:41:08.368539 7916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.125@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
I20260927 22:41:08.380107 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.369190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57222)
0927 22:41:08.369368 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.369387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.369503 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.369897 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.369901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.370229 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:08.370340 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.370347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.371259 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.371266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.372095 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.372105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.372778 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.372783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.372952 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.373410 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:08.373433 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:08.375327 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:08.377932 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.377937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.377941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.378271 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.378603 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.378609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.378613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.378728 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.379201 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:08.379206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:08.379331 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.379460 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.379741 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:41:08.380178 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.369392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.1.53.125:37173)
0927 22:41:08.369537 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.369545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.369562 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:41:08.369728 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.369731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.369781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.369884 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.370468 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.371141 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.372225 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.372645 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.372774 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.375505 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.375509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.375513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.375536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.377801 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.378378 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.378386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.378389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.378502 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.378839 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.378843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.378846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.379094 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.379235 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.379535 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.379877 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":57}
I20260927 22:41:08.384663 7916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Bootstrap starting.
I20260927 22:41:08.387622 7916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:08.388700 7916 log.cc:824] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:08.391599 7916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: No bootstrap required, opened a new log
I20260927 22:41:08.399540 7916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } }
I20260927 22:41:08.399909 7916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:08.400049 7916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f90e4b26e334500ba8d3b211087d65c, State: Initialized, Role: FOLLOWER
I20260927 22:41:08.400723 7916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } }
I20260927 22:41:08.401630 7923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } } }
I20260927 22:41:08.401958 7923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:08.402755 7916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:41:08.411623 7934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:41:08.411755 7934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:41:08.414096 7849 master_runner.cc:428] Detected that this master 2f90e4b26e334500ba8d3b211087d65c is joining an existing cluster
I20260927 22:41:08.414197 7849 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.53.125:37173
I20260927 22:41:08.436885 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.417966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.1.53.125:37173)
0927 22:41:08.418153 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.418162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.419496 (+ 1334us) server_negotiation.cc:407] Connection header received
0927 22:41:08.419680 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.419687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.419799 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.419886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.421632 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.422906 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.425416 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.426260 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.426370 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.429667 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.429675 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.429681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.429717 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.433860 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.434709 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.434714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.434718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.434848 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.435248 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.435252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.435255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.435583 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.435688 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.436128 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.436402 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:41:08.437266 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.418092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49004)
0927 22:41:08.419002 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.419031 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.419242 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.419606 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.419609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.420408 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:08.420932 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.421275 (+ 343us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.422916 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.422924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.425293 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.425310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.426726 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.426736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.426988 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.428028 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:08.428055 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:08.429522 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:08.433909 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.433915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.433919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.434550 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.435045 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.435051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.435056 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.435188 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.435878 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:08.435881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:08.436074 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.436217 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.436571 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":160,"thread_start_us":555,"threads_started":1}
I20260927 22:41:08.437325 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.418011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.1.53.126:42075)
0927 22:41:08.418215 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.418224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.419124 (+ 900us) server_negotiation.cc:407] Connection header received
0927 22:41:08.419380 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.419386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.419494 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.419589 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.421455 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.422625 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.425463 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.426234 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.426376 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.429798 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.429807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.429813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.429859 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.433751 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.434728 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.434736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.434740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.434927 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.435319 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.435325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.435328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.435732 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.435914 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.436423 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.436722 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:41:08.437863 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.418314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57234)
0927 22:41:08.419377 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.419397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.419538 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.420096 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.420102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.421073 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:08.421461 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.421474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.423199 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.423205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.425230 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.425249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.426802 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.426820 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.427124 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.427904 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:08.427930 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:08.429436 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:08.433990 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.433997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.434003 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.434588 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.434963 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.434969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.434972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.435137 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.435734 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:08.435737 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:08.435875 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.437092 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.437417 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":144,"thread_start_us":747,"threads_started":1}
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548866, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.126@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.124@KRBTEST.COM" created.
I20260927 22:41:08.463367 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.444191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.1.53.125:37173)
0927 22:41:08.444418 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.444428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.448950 (+ 4522us) server_negotiation.cc:407] Connection header received
0927 22:41:08.449309 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.449313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.449404 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.449513 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.450618 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.451357 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.453125 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.453842 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.453975 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.457663 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.457668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.457671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.457701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.460539 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.461229 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.461238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.461241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.461420 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.461819 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.461824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.461831 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.462097 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.462251 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.462768 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.463005 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:41:08.463768 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.444119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57242)
0927 22:41:08.448817 (+ 4698us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.448866 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.449087 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.449576 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.449585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.450234 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:08.450443 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.450452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.451507 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.451512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.452983 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.452997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.454012 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.454018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.454245 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.455105 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:08.455122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:08.457470 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:08.460680 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.460691 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.460700 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.461098 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.461553 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.461559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.461563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.461706 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.462243 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:08.462253 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:08.462463 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.462671 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.463005 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4467,"thread_start_us":4317,"threads_started":1}
I20260927 22:41:08.465775 7387 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.53.125:37173: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c"
member_type: NON_VOTER
last_known_addr {
host: "127.1.53.125"
port: 37173
}
attrs {
promote: true
}
}
dest_uuid: "057aa11ad898437fa599b88a33ccb839"
cas_config_opid_index: -1
I20260927 22:41:08.466755 7387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } }
I20260927 22:41:08.471974 7946 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:08.472888 7891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:08.475526 7891 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:41:08.476207 7946 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.53.125:37173
I20260927 22:41:08.476953 7849 master.cc:537] Master@127.1.53.125:37173 shutting down...
I20260927 22:41:08.487128 7948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:08.491067 7849 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:08.491480 7849 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:08.491681 7849 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: stopping tablet replica
W20260927 22:41:08.494757 7374 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 22:41:08.502192 7849 master.cc:552] Master@127.1.53.125:37173 shutdown complete.
I20260927 22:41:08.502285 7849 master_runner.cc:305] Clearing existing system tablet
I20260927 22:41:08.502714 7849 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:08.502812 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:08.509193 7945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:08.509648 7945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:08.510660 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.510017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57254)
0927 22:41:08.510271 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.510415 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 22:41:08.511189 7374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 -> Peer 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125:37173): Couldn't send request to peer 2f90e4b26e334500ba8d3b211087d65c. Status: Network error: Client connection negotiation failed: client connection to 127.1.53.125:37173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:41:08.512584 7849 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.003s
I20260927 22:41:08.514892 7953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:08.515260 7849 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:08.515537 7849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:41:08.515635 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
uuid: "2f90e4b26e334500ba8d3b211087d65c"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "1dc2c97d36d8c723ad2108238ae09b85"
server_key_iv: "7f972afab11e1748a06a64a4bad47b21"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.124.keytab.
Entry for principal oryx/127.1.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.124.keytab.
I20260927 22:41:08.517835 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:36011
--webserver_interface=127.1.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.124.keytab
--principal=oryx/127.1.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:41:08.521003 7849 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:08.527061 7849 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:41:08.527257 7849 log.cc:1196] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:41:08.527633 7849 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Deleting consensus metadata
I20260927 22:41:08.527890 7849 master_runner.cc:315] Copying system tablet from 127.1.53.126:42075
I20260927 22:41:08.530447 7849 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: tablet copy: Beginning tablet copy session from remote peer at address 127.1.53.126:42075
I20260927 22:41:08.542620 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.531199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.1.53.126:42075)
0927 22:41:08.531402 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.531409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.531772 (+ 363us) server_negotiation.cc:407] Connection header received
0927 22:41:08.531991 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.531997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.532108 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.532221 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:08.533057 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.533792 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.535021 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.535430 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.535583 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.537427 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:08.537433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:08.537440 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:08.537476 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:08.540071 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.540706 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.540714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.540717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.540840 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:08.541229 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:08.541234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:08.541238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:08.541504 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:08.541659 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.542032 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.542227 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:41:08.542659 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.531168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49010)
0927 22:41:08.531645 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.531672 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.531844 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.532272 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.532276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.532686 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:08.532894 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.532903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.533936 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.533942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.534869 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.534880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.535603 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.535609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.535812 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.536513 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:08.536529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:08.537234 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:08.540201 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.540208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.540212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.540586 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.540959 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:08.540967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:08.540971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:08.541115 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:08.541621 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:08.541627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:08.541792 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.541930 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.542239 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":255,"threads_started":1}
I20260927 22:41:08.543787 7407 tablet_copy_service.cc:140] P 057aa11ad898437fa599b88a33ccb839: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2f90e4b26e334500ba8d3b211087d65c ({username='oryx', principal='oryx/127.1.53.125@KRBTEST.COM'} at 127.0.0.1:49010)
I20260927 22:41:08.544070 7407 tablet_copy_service.cc:161] P 057aa11ad898437fa599b88a33ccb839: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2f90e4b26e334500ba8d3b211087d65c at {username='oryx', principal='oryx/127.1.53.125@KRBTEST.COM'} at 127.0.0.1:49010: session id = 2f90e4b26e334500ba8d3b211087d65c-00000000000000000000000000000000
I20260927 22:41:08.547250 7407 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:41:08.549332 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:41:08.554190 7849 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: tablet copy: Starting download of 0 data blocks...
I20260927 22:41:08.554498 7849 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: tablet copy: Starting download of 1 WAL segments...
I20260927 22:41:08.558420 7849 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:41:08.561681 7407 tablet_copy_service.cc:342] P 057aa11ad898437fa599b88a33ccb839: Request end of tablet copy session 2f90e4b26e334500ba8d3b211087d65c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.53.125@KRBTEST.COM'} at 127.0.0.1:49010
I20260927 22:41:08.561880 7407 tablet_copy_service.cc:433] P 057aa11ad898437fa599b88a33ccb839: ending tablet copy session 2f90e4b26e334500ba8d3b211087d65c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2f90e4b26e334500ba8d3b211087d65c
I20260927 22:41:08.572407 7849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:08.575260 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:08.577991 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:08.588479 7964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:08.593537 7849 server_base.cc:1034] running on GCE node
I20260927 22:41:08.593918 7849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:08.594548 7849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:08.595772 7849 hybrid_clock.cc:648] HybridClock initialized: now 1790548868595697 us; error 53 us; skew 500 ppm
I20260927 22:41:08.596009 7849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:08.598057 7849 init.cc:377] Logged in from keytab as oryx/127.1.53.125@KRBTEST.COM (short username oryx)
I20260927 22:41:08.599470 7849 webserver.cc:460] Webserver started at http://127.1.53.125:37389/ using document root <none> and password file <none>
I20260927 22:41:08.599795 7849 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:08.599872 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:08.602262 7849 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 22:41:08.603850 7971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:08.604282 7849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 22:41:08.604435 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
uuid: "2f90e4b26e334500ba8d3b211087d65c"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "1dc2c97d36d8c723ad2108238ae09b85"
server_key_iv: "7f972afab11e1748a06a64a4bad47b21"
server_key_version: "encryptionkey@0"
I20260927 22:41:08.604594 7849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:08.627511 7849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:08.631089 7849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:08.659358 7849 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.125:37173
I20260927 22:41:08.659377 8022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.125:37173 every 8 connection(s)
I20260927 22:41:08.660327 7849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
I20260927 22:41:08.667884 8023 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:41:08.670440 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Bootstrap starting.
I20260927 22:41:08.682471 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:41:08.682936 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: Bootstrap complete.
I20260927 22:41:08.683780 8023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } }
I20260927 22:41:08.684011 8023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f90e4b26e334500ba8d3b211087d65c, State: Initialized, Role: LEARNER
I20260927 22:41:08.684299 8023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } }
I20260927 22:41:08.685092 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:08.685411 8023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:08.685465 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: LEARNER
I20260927 22:41:08.694674 8037 catalog_manager.cc:1259] Loaded cluster ID: cadfcac557c94fdeb02e19f5913049bd
I20260927 22:41:08.694798 8037 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: loading cluster ID for follower catalog manager: success
I20260927 22:41:08.697191 8037 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: acquiring CA information for follower catalog manager: success
I20260927 22:41:08.698927 8037 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:41:08.846568 7844 heartbeater.cc:503] Master 127.1.53.126:42075 was elected leader, sending a full tablet report...
W20260927 22:41:08.941614 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:08.941900 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:08.941987 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:08.949710 7954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:08.949812 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:08.949913 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:08.950009 7954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:08.950070 7954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:08.961030 7954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.124.keytab
--principal=oryx/127.1.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:08.962422 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:08.963727 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:08.978221 8043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:08.978411 8044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:08.978544 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.967641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57260)
0927 22:41:08.967842 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:08.967865 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:08.968007 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:08.971970 (+ 3963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:08.971975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:08.972020 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:08.972209 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.972220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.973708 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.973714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.975758 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:08.975776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.977682 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.977690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:08.977916 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.978009 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 22:41:08.978098 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:41:08.978260 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:41:08.980002 8046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:08.980201 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:08.967897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.1.53.125:37173)
0927 22:41:08.971629 (+ 3732us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:08.971634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:08.971651 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:41:08.971712 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:08.971715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:08.971784 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:08.971889 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:08.972397 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.973465 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.975997 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:08.977491 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:08.978911 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:08.979610 (+ 699us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:08.979664 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 22:41:08.979822 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3623,"thread_start_us":3502,"threads_started":1}
I20260927 22:41:08.980171 7954 server_base.cc:1034] running on GCE node
I20260927 22:41:08.981016 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:08.981222 7998 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 22:41:08.981753 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057aa11ad898437fa599b88a33ccb839. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:08.981985 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: LEARNER
I20260927 22:41:08.982712 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:08.983942 7954 hybrid_clock.cc:648] HybridClock initialized: now 1790548868983905 us; error 47 us; skew 500 ppm
I20260927 22:41:08.984338 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:08 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:08.987448 7954 init.cc:377] Logged in from keytab as oryx/127.1.53.124@KRBTEST.COM (short username oryx)
I20260927 22:41:08.989100 7954 webserver.cc:460] Webserver started at http://127.1.53.124:44055/ using document root <none> and password file <none>
I20260927 22:41:08.989780 7954 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:08.989920 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:08.990216 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:08.993149 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/instance:
uuid: "7548b756336348acbf16385a39e73add"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "d6015672dae2bf62c3e73a6786aeb1a8"
server_key_iv: "83c4e86504b6ad1d6ebba00d71070799"
server_key_version: "encryptionkey@0"
I20260927 22:41:08.993943 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal/instance:
uuid: "7548b756336348acbf16385a39e73add"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "d6015672dae2bf62c3e73a6786aeb1a8"
server_key_iv: "83c4e86504b6ad1d6ebba00d71070799"
server_key_version: "encryptionkey@0"
I20260927 22:41:08.999248 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:41:09.003091 8053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:09.004567 7954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:09.004746 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
uuid: "7548b756336348acbf16385a39e73add"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "d6015672dae2bf62c3e73a6786aeb1a8"
server_key_iv: "83c4e86504b6ad1d6ebba00d71070799"
server_key_version: "encryptionkey@0"
I20260927 22:41:09.004954 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:09.023701 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:09.028065 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:09.028414 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:09.048175 7954 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.124:36011
I20260927 22:41:09.048194 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.124:36011 every 8 connection(s)
I20260927 22:41:09.049613 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
I20260927 22:41:09.053792 8105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:09.057477 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 7954
I20260927 22:41:09.063531 8105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.124@KRBTEST.COM
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.124@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:09.083930 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.067318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.1.53.126:42075)
0927 22:41:09.068088 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.068094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.068113 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:41:09.068173 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.068179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.068283 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.068399 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.069766 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.070649 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.072331 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.073079 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.073217 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.077887 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.077893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.077898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.077937 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.081038 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.081842 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.081848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.081851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.081975 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.082408 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.082414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.082417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.082781 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.082923 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.083513 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.083669 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":549,"threads_started":1}
I20260927 22:41:09.084821 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.066549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49026)
0927 22:41:09.067575 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.067647 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.067863 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.068490 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.068498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.069242 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.069577 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.069595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.070808 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.070812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.072146 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.072159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.073314 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.073319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.073441 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.074244 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.074283 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.077642 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.081167 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.081175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.081181 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.081721 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.082098 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.082102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.082105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.082275 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.082929 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.082936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.083153 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.083362 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.083652 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":2,"thread_start_us":509,"threads_started":1}
I20260927 22:41:09.084841 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.060016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.1.53.124:36011)
0927 22:41:09.060936 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.060948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.060991 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:41:09.062971 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.062995 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.063448 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.063908 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.065536 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.067573 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.069211 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.070163 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.070304 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.076162 (+ 5858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.076180 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.076205 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.076299 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.079725 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.080735 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.080743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.080751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.080907 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.081349 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.081354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.081357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.082091 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.082411 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.082810 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.083306 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":683,"thread_start_us":444,"threads_started":1}
I20260927 22:41:09.086781 8105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.124@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
I20260927 22:41:09.099973 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.087709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.1.53.125:37173)
0927 22:41:09.087912 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.087919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.088135 (+ 216us) server_negotiation.cc:407] Connection header received
0927 22:41:09.088368 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.088373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.088455 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.088557 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.089205 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.089921 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.091131 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.091560 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.091688 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.094910 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.094918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.094928 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.094958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.097577 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.098185 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.098191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.098194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.098323 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.098668 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.098675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.098678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.098933 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.099081 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.099435 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.099658 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:41:09.099973 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.087723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57272)
0927 22:41:09.088030 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.088049 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.088219 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.088576 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.088581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.088941 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.089067 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.089074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.090063 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.090071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.090997 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.091007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.091710 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.091716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.091887 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.092467 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.092483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.094729 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.097706 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.097712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.097716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.098070 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.098436 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.098441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.098444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.098568 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.099045 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.099049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.099198 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.099331 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.099614 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 22:41:09.101053 8105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.124@KRBTEST.COM for oryx/127.1.53.124@KRBTEST.COM
I20260927 22:41:09.113627 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.101852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.1.53.124:36011)
0927 22:41:09.101976 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.101981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.101999 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:41:09.102048 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.102053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.102111 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.102202 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.102796 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.103447 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.104568 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.104967 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.105099 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.108275 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.108282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.108293 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.108322 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.111058 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.111698 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.111703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.111713 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.111820 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.112190 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.112197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.112200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.112528 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.112667 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.113028 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.113327 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":42}
I20260927 22:41:09.113646 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.101705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.0.0.1:44356)
0927 22:41:09.101845 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.101859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.101968 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.102216 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.102220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.102561 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.102676 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.102684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.103585 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.103589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.104447 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.104457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.105103 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.105108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.105292 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.105919 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.105936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.108046 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.111215 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.111222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.111226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.111574 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.111924 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.111930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.111936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.112086 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.112643 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.112648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.112777 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.112918 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.113197 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:41:09.118309 8105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Bootstrap starting.
I20260927 22:41:09.121816 8105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:09.122972 8105 log.cc:824] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:09.125785 8105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: No bootstrap required, opened a new log
I20260927 22:41:09.132892 8105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } }
I20260927 22:41:09.133340 8105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:09.133445 8105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7548b756336348acbf16385a39e73add, State: Initialized, Role: FOLLOWER
I20260927 22:41:09.133937 8105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } }
I20260927 22:41:09.134689 8112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } } }
I20260927 22:41:09.135185 8112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:09.135910 8105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:41:09.146016 8123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:41:09.146191 8123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:41:09.150818 7954 master_runner.cc:428] Detected that this master 7548b756336348acbf16385a39e73add is joining an existing cluster
I20260927 22:41:09.150934 7954 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.53.124:36011
I20260927 22:41:09.173586 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.154838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.1.53.126:42075)
0927 22:41:09.155049 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.155055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.156494 (+ 1439us) server_negotiation.cc:407] Connection header received
0927 22:41:09.156763 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.156769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.156854 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.156923 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.158464 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.159662 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.162003 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.162646 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.162750 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.165804 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.165830 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.165835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.165877 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.169741 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.170971 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.170976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.170980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.171139 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.171631 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.171636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.171639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.171968 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.172108 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.172981 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.173204 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:41:09.174855 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.155047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49040)
0927 22:41:09.156392 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.156411 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.156610 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.157132 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.157137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.157918 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.158270 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.158283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.159845 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.159850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.161839 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.161854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.163017 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.163024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.163205 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.164039 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.164072 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.165602 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.170334 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.170339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.170343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.170826 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.171273 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.171278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.171282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.171482 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.172167 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.172172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.172405 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.172618 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.174349 (+ 1731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"spinlock_wait_cycles":8576,"thread_start_us":996,"threads_started":1}
I20260927 22:41:09.183007 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.157112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.1.53.125:37173)
0927 22:41:09.157347 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.157356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.167249 (+ 9893us) server_negotiation.cc:407] Connection header received
0927 22:41:09.167497 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.167504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.167582 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.167651 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.169129 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.170178 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.172202 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.173358 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.173465 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.176341 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.176348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.176355 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.176400 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.180038 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.180802 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.180809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.180812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.180955 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.181436 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.181441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.181444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.181767 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.181884 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.182461 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.182692 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:41:09.183938 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.157729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:57280)
0927 22:41:09.167151 (+ 9422us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.167169 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.167339 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.167854 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.167858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.168509 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.168945 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.168961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.170367 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.170374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.172024 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.172043 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.173761 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.173770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.173958 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.174610 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.174631 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.176100 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.180171 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.180178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.180181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.180679 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.181105 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.181109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.181112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.181310 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.181959 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.181963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.182181 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.183167 (+ 986us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.183468 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9297,"mutex_wait_us":49,"thread_start_us":8021,"threads_started":1}
I20260927 22:41:09.186051 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.155181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.1.53.124:36011)
0927 22:41:09.155365 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.155371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.164042 (+ 8671us) server_negotiation.cc:407] Connection header received
0927 22:41:09.164228 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.164234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.164316 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.164405 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.166936 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.168112 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.170458 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.171233 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.171328 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.174366 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.174376 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.174385 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.174424 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.178798 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.179561 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.179567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.179571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.179697 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.180112 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.180116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.180119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.180528 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.180653 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.185408 (+ 4755us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.185664 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:41:09.187609 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.155551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.0.0.1:44372)
0927 22:41:09.163951 (+ 8400us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.163970 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.164119 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.164637 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.164642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.165313 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.166780 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.166793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.168664 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.168669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.170322 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.170335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.171627 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.171633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.171854 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.172535 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.172558 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.174172 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.178940 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.178946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.178952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.179440 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.179821 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.179825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.179828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.179977 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.180734 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.180738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.180918 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.186841 (+ 5923us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.187175 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8232,"mutex_wait_us":31,"thread_start_us":10575,"threads_started":1}
I20260927 22:41:09.192368 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7448
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548866, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.126@KRBTEST.COM for oryx/127.1.53.124@KRBTEST.COM
I20260927 22:41:09.204715 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.180705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.0.0.1:44386)
0927 22:41:09.181109 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.181129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.181231 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.186706 (+ 5475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.186713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.187196 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.187443 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.187463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.189351 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.189358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.190820 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.190835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.193233 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.193240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.193405 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.194132 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.194155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.196949 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.201946 (+ 4997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.201952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.201956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.202428 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.202919 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.202925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.202929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.203089 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.203694 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.203700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.203853 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.203994 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.204286 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260927 22:41:09.205778 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.181427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.1.53.124:36011)
0927 22:41:09.186322 (+ 4895us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.186328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.186344 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:41:09.186438 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.186443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.186557 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.186654 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.187654 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.189170 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.192381 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.193045 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.193750 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.197166 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.197172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.197177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.197213 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.201780 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.202591 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.202597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.202601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.202763 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.203217 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.203222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.203226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.203568 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.205005 (+ 1437us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.205128 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.205366 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4794,"threads_started":1}
I20260927 22:41:09.209116 7387 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.53.124:36011: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7548b756336348acbf16385a39e73add"
member_type: NON_VOTER
last_known_addr {
host: "127.1.53.124"
port: 36011
}
attrs {
promote: true
}
}
dest_uuid: "057aa11ad898437fa599b88a33ccb839"
cas_config_opid_index: 5
I20260927 22:41:09.210152 7387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } }
I20260927 22:41:09.212457 7998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 LEARNER]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:41:09.213197 7946 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7548b756336348acbf16385a39e73add (127.1.53.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.213742 8134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:41:09.214063 8080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:09.216847 8080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:41:09.216575 7998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7548b756336348acbf16385a39e73add (127.1.53.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.217710 8134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:09.222240 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.222599 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: LEARNER
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.65@KRBTEST.COM
I20260927 22:41:09.227190 8080 raft_consensus.cc:858] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } }
W20260927 22:41:09.227397 8080 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } }
I20260927 22:41:09.235402 7946 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.53.124:36011
I20260927 22:41:09.235774 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.194127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.1.53.65:44621)
0927 22:41:09.206625 (+ 12498us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.206632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.206659 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:41:09.206727 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.206734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.206942 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.207149 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.208865 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.211004 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.219271 (+ 8267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.220729 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.220883 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.227807 (+ 6924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.227822 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.227838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.227899 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.231940 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.232887 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.232898 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.232902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.233059 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.233520 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.233525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.233528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.233972 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.234237 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.235078 (+ 841us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.235352 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12350,"thread_start_us":12199,"threads_started":1}
I20260927 22:41:09.236344 7946 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: attempting to promote NON_VOTER 2f90e4b26e334500ba8d3b211087d65c to VOTER
I20260927 22:41:09.236312 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.237183 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.237906 7946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } }
I20260927 22:41:09.238507 7571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:09.244503 8136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.245620 8136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.250468 7998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 LEARNER]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:41:09.251263 8142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:41:09.254346 8134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.255227 7998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.259270 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.259542 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.259851 8136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.259862 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.260087 8136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.260113 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:09.264815 7954 master.cc:537] Master@127.1.53.124:36011 shutting down...
I20260927 22:41:09.281646 8080 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:09.287429 8079 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260927 22:41:09.287361 8146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:09.287632 8146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 22:41:09.287706 8079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:41:09.287827 8112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057aa11ad898437fa599b88a33ccb839. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:09.288035 8112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 22:41:09.287461 8143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: NON_VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: true } } }
I20260927 22:41:09.289382 8143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 22:41:09.289710 8142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:41:09.291131 7954 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 22:41:09.291581 7954 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 22:41:09.291741 7954 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: stopping tablet replica
I20260927 22:41:09.305841 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.305325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.0.0.1:44398)
0927 22:41:09.305522 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.305617 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:41:09.306102 7372 consensus_peers.cc:487] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 -> Peer 7548b756336348acbf16385a39e73add (127.1.53.124:36011): Couldn't send request to peer 7548b756336348acbf16385a39e73add. Status: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:41:09.309716 7954 master.cc:552] Master@127.1.53.124:36011 shutdown complete.
I20260927 22:41:09.309808 7954 master_runner.cc:305] Clearing existing system tablet
I20260927 22:41:09.310226 7954 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:09.310333 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:09.313277 7954 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 22:41:09.314951 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:09.315508 7954 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:09.315780 7954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 22:41:09.315884 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
uuid: "7548b756336348acbf16385a39e73add"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "d6015672dae2bf62c3e73a6786aeb1a8"
server_key_iv: "83c4e86504b6ad1d6ebba00d71070799"
server_key_version: "encryptionkey@0"
I20260927 22:41:09.320600 7954 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:09.328042 7954 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:41:09.328217 7954 log.cc:1196] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:41:09.328799 7954 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Deleting consensus metadata
I20260927 22:41:09.329047 7954 master_runner.cc:315] Copying system tablet from 127.1.53.126:42075
I20260927 22:41:09.331570 7954 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: tablet copy: Beginning tablet copy session from remote peer at address 127.1.53.126:42075
I20260927 22:41:09.351267 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.332335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49044)
0927 22:41:09.336368 (+ 4033us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.336394 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.336579 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.337027 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.337032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.337575 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.337831 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.337844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.339276 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.339282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.340866 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.340881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.341851 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.341858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.342056 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.342831 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.342854 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.344430 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.348386 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.348393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.348397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.348891 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.349322 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.349327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.349331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.349532 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.350134 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.350139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.350337 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.350484 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.350816 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3822,"thread_start_us":3729,"threads_started":1}
I20260927 22:41:09.352290 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.332427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.1.53.126:42075)
0927 22:41:09.332657 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.332664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.336543 (+ 3879us) server_negotiation.cc:407] Connection header received
0927 22:41:09.336737 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.336743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.336827 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.336948 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.337978 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.339109 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.341012 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.341692 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.341816 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.344642 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.344649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.344654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.344694 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.348192 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.349050 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.349056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.349060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.349206 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.349668 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.349674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.349677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.350012 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.350151 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.351668 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.351895 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:41:09.352838 7407 tablet_copy_service.cc:140] P 057aa11ad898437fa599b88a33ccb839: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7548b756336348acbf16385a39e73add ({username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:49044)
I20260927 22:41:09.353004 7407 tablet_copy_service.cc:161] P 057aa11ad898437fa599b88a33ccb839: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7548b756336348acbf16385a39e73add at {username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:49044: session id = 7548b756336348acbf16385a39e73add-00000000000000000000000000000000
I20260927 22:41:09.356673 7407 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:41:09.358631 7954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:41:09.361405 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7448
I20260927 22:41:09.367098 7954 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: tablet copy: Starting download of 0 data blocks...
I20260927 22:41:09.367372 7954 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: tablet copy: Starting download of 1 WAL segments...
I20260927 22:41:09.369525 7954 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:41:09.372637 7407 tablet_copy_service.cc:342] P 057aa11ad898437fa599b88a33ccb839: Request end of tablet copy session 7548b756336348acbf16385a39e73add-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:49044
I20260927 22:41:09.372779 7407 tablet_copy_service.cc:433] P 057aa11ad898437fa599b88a33ccb839: ending tablet copy session 7548b756336348acbf16385a39e73add-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7548b756336348acbf16385a39e73add
I20260927 22:41:09.382752 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:41:09.384218 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:44621
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:42075
--webserver_port=42837
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:09.386590 8156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:09.392602 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:09.404548 7954 server_base.cc:1034] running on GCE node
W20260927 22:41:09.405819 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:09.406358 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:09.407137 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:09.408420 7954 hybrid_clock.cc:648] HybridClock initialized: now 1790548869408378 us; error 30 us; skew 500 ppm
I20260927 22:41:09.408741 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:09.411185 7954 init.cc:377] Logged in from keytab as oryx/127.1.53.124@KRBTEST.COM (short username oryx)
I20260927 22:41:09.412662 7954 webserver.cc:460] Webserver started at http://127.1.53.124:34131/ using document root <none> and password file <none>
I20260927 22:41:09.413035 7954 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:09.413151 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:09.416431 7954 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:41:09.418371 8167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:09.418886 7954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 22:41:09.419010 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
uuid: "7548b756336348acbf16385a39e73add"
format_stamp: "Formatted at 2026-09-27 22:41:08 on dist-test-slave-pp80"
server_key: "d6015672dae2bf62c3e73a6786aeb1a8"
server_key_iv: "83c4e86504b6ad1d6ebba00d71070799"
server_key_version: "encryptionkey@0"
I20260927 22:41:09.419200 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:09.438308 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:09.442408 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:09.478083 7954 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.124:36011
I20260927 22:41:09.478286 8218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.124:36011 every 8 connection(s)
I20260927 22:41:09.479115 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
I20260927 22:41:09.484562 8219 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:41:09.489193 8219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Bootstrap starting.
I20260927 22:41:09.504894 8219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:41:09.505417 8219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Bootstrap complete.
I20260927 22:41:09.506527 8219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } }
I20260927 22:41:09.506863 8219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7548b756336348acbf16385a39e73add, State: Initialized, Role: LEARNER
I20260927 22:41:09.507119 8219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } }
I20260927 22:41:09.512756 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.513088 8219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:09.513238 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: LEARNER
I20260927 22:41:09.531839 8233 catalog_manager.cc:1259] Loaded cluster ID: cadfcac557c94fdeb02e19f5913049bd
I20260927 22:41:09.531949 8233 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: loading cluster ID for follower catalog manager: success
I20260927 22:41:09.535329 8233 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: acquiring CA information for follower catalog manager: success
I20260927 22:41:09.537765 8233 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:41:09.766331 8159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:09.766674 8159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:09.766834 8159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:09.774487 8159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:09.774636 8159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:09.774802 8159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:09.784621 8159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:44621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=42837
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:09.786114 8159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:09.787500 8159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:09.802014 8238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:09.802089 8241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:09.802095 8239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:09.803090 8159 server_base.cc:1034] running on GCE node
I20260927 22:41:09.803632 8159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:09.804529 8159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:09.805783 8159 hybrid_clock.cc:648] HybridClock initialized: now 1790548869805694 us; error 67 us; skew 500 ppm
I20260927 22:41:09.806170 8159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:09.810137 8159 init.cc:377] Logged in from keytab as oryx/127.1.53.65@KRBTEST.COM (short username oryx)
I20260927 22:41:09.811741 8159 webserver.cc:460] Webserver started at http://127.1.53.65:42837/ using document root <none> and password file <none>
I20260927 22:41:09.812572 8159 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:09.812728 8159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:09.814330 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.806886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.0.0.1:44400)
0927 22:41:09.807363 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.807389 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.807562 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.808394 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.808398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.808422 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:09.808604 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.808611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.810062 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.810071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.811962 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.811980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.813268 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.813275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.813491 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.813528 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.813618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.813753 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":576,"threads_started":1}
I20260927 22:41:09.814503 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.806979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.1.53.124:36011)
0927 22:41:09.808007 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.808015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.808036 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:41:09.808110 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.808116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.808208 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.808358 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:09.808754 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.809912 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.812109 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.813083 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.813193 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.813419 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.813662 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.813756 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":726,"threads_started":1}
I20260927 22:41:09.815455 8194 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260927 22:41:09.815883 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057aa11ad898437fa599b88a33ccb839. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: NON_VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: true } } }
I20260927 22:41:09.816143 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: LEARNER
I20260927 22:41:09.817008 8142 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: attempting to promote NON_VOTER 7548b756336348acbf16385a39e73add to VOTER
I20260927 22:41:09.817579 8142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:09.819725 8194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 LEARNER]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:41:09.820182 7998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Refusing update from remote peer 057aa11ad898437fa599b88a33ccb839: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:41:09.821075 8249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:41:09.821892 8249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:41:09.825847 8249 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7548b756336348acbf16385a39e73add (127.1.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.826210 8194 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7548b756336348acbf16385a39e73add (127.1.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.830812 8142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.831099 8142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.830816 8249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.831708 8249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.832402 8222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.832659 8222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:09.832844 8249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.833096 8249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:09.833603 8159 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.009s sys 0.007s
I20260927 22:41:09.833523 7998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7548b756336348acbf16385a39e73add (127.1.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.836745 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:09.836773 8256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "057aa11ad898437fa599b88a33ccb839" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:09.837132 8256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:09.838333 8159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:41:09.838527 8159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "ded01630111646d1a7fa7d6cc9c12b7a"
format_stamp: "Formatted at 2026-09-27 22:41:06 on dist-test-slave-pp80"
server_key: "5d279f7a74ba5091d21659691838d5d5"
server_key_iv: "5883e0c6fff80efa994624f51fff4786"
server_key_version: "encryptionkey@0"
I20260927 22:41:09.839515 8159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:09.851408 8159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:09.856094 8159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:09.856395 8159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:09.857386 8159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:09.858817 8159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:09.858933 8159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:09.859067 8159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:09.859153 8159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:09.893457 8159 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:44621
I20260927 22:41:09.893513 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:44621 every 8 connection(s)
I20260927 22:41:09.894927 8159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:09.905131 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 8159
I20260927 22:41:09.905573 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7582
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.66@KRBTEST.COM
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.124@KRBTEST.COM
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
Sep 27 22:41:09 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
I20260927 22:41:09.943300 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.906971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.1.53.66:41405)
0927 22:41:09.916220 (+ 9249us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.916233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.917212 (+ 979us) server_negotiation.cc:407] Connection header received
0927 22:41:09.917309 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.917322 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.917700 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.917863 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.920492 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.922588 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.924607 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.925447 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.925571 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.933439 (+ 7868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.933455 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.933474 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.933524 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.937260 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.938179 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.938190 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.938197 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.938356 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.938802 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.938808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.938813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.939264 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.939578 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.940789 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.941052 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9068,"thread_start_us":8906,"threads_started":1}
I20260927 22:41:09.943336 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.898165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:55181 (local address 127.1.53.124:36011)
0927 22:41:09.898490 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.898497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.911175 (+ 12678us) server_negotiation.cc:407] Connection header received
0927 22:41:09.914695 (+ 3520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.914704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.914815 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.914943 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.922664 (+ 7721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.923776 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.926711 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.927350 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.927454 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.934019 (+ 6565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.934027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.934033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.934078 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.938842 (+ 4764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.939906 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.939911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.939915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.940067 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.940625 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.940630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.940633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.940963 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.941089 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.942300 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.942996 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260927 22:41:09.943454 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.898250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:54811 (local address 127.1.53.126:42075)
0927 22:41:09.918410 (+ 20160us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.918416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.919056 (+ 640us) server_negotiation.cc:407] Connection header received
0927 22:41:09.919144 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.919152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.919253 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.919386 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.922683 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.923973 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.926705 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.927350 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.927487 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.935085 (+ 7598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.935091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.935095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.935138 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.938842 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.939906 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.939911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.939915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.940067 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.940627 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.940632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.940635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.940963 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.941090 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.942361 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.943058 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20049,"thread_start_us":19963,"threads_started":1}
I20260927 22:41:09.943988 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.898167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:51223 (local address 127.1.53.125:37173)
0927 22:41:09.919923 (+ 21756us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:09.919930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:09.919951 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:41:09.920009 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:09.920015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:09.920090 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:09.920211 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:09.922650 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.923921 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.926659 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.927313 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.927447 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.935102 (+ 7655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:09.935110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:09.935116 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:09.935159 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:09.938652 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.939913 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.939919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.939925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.940084 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:09.940656 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:09.940660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:09.940663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:09.940996 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:09.941152 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:09.942910 (+ 1758us) server_negotiation.cc:299] Negotiation successful
0927 22:41:09.943289 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21618,"thread_start_us":21462,"threads_started":1}
I20260927 22:41:09.945888 7705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:09.950299 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.898642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:54811)
0927 22:41:09.918873 (+ 20231us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.918892 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.919025 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.920343 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.920353 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.921308 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.922410 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.922439 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.924346 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.924353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.926532 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.926546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.927783 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.927790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.928354 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.929277 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.929294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.934855 (+ 5561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.938998 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.939003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.939008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.939781 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.940196 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.940201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.940205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.940496 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.941181 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.941191 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.941764 (+ 573us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.943605 (+ 1841us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.944202 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20078,"mutex_wait_us":1167,"spinlock_wait_cycles":986752,"thread_start_us":19403,"threads_started":1}
I20260927 22:41:09.950331 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.898641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.1.53.65:51223)
0927 22:41:09.915240 (+ 16599us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.915268 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.915408 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.920223 (+ 4815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.920226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.921202 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.922465 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.922476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.924086 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.924096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.926460 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.926477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.927846 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.927855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.928347 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.929517 (+ 1170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.929558 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.934859 (+ 5301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.938828 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.938837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.938858 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.939773 (+ 915us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.940216 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.940223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.940228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.940514 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.941214 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.941222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.941739 (+ 517us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.942148 (+ 409us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.942677 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16402,"mutex_wait_us":377,"spinlock_wait_cycles":79616,"thread_start_us":11484,"threads_started":1}
I20260927 22:41:09.950299 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:09.898641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.1.53.65:55181)
0927 22:41:09.910923 (+ 12282us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:09.911000 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:09.914315 (+ 3315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:09.920057 (+ 5742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:09.920088 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:09.921202 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:09.922421 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.922443 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.923998 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.924005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.926531 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:09.926546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:09.927783 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:09.927790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:09.928355 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:09.929219 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:09.929257 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:09.933744 (+ 4487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:09.938998 (+ 5254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.939003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.939008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.939757 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.940201 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:09.940205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:09.940208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:09.940502 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:09.941192 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:09.941195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:09.941774 (+ 579us) client_negotiation.cc:769] Sending connection context
0927 22:41:09.943597 (+ 1823us) client_negotiation.cc:240] Negotiation successful
0927 22:41:09.943992 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11814,"mutex_wait_us":654,"spinlock_wait_cycles":19072,"thread_start_us":16702,"threads_started":1}
I20260927 22:41:09.952131 8369 heartbeater.cc:347] Connected to a master server at 127.1.53.124:36011
I20260927 22:41:09.952154 8370 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42075
I20260927 22:41:09.952363 8371 heartbeater.cc:347] Connected to a master server at 127.1.53.125:37173
I20260927 22:41:09.952553 8371 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:09.952554 8369 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:09.952557 8370 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:09.953708 8370 heartbeater.cc:511] Master 127.1.53.126:42075 requested a full tablet report, sending...
I20260927 22:41:09.953778 8369 heartbeater.cc:511] Master 127.1.53.124:36011 requested a full tablet report, sending...
I20260927 22:41:09.953751 8371 heartbeater.cc:511] Master 127.1.53.125:37173 requested a full tablet report, sending...
I20260927 22:41:09.956050 8184 ts_manager.cc:194] Registered new tserver with Master: ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65:44621)
I20260927 22:41:09.956657 7988 ts_manager.cc:194] Registered new tserver with Master: ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65:44621)
I20260927 22:41:09.957129 7387 ts_manager.cc:194] Re-registered known tserver with Master: ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65:44621)
I20260927 22:41:09.958092 7387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:54811
I20260927 22:41:10.037756 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7582
I20260927 22:41:10.054423 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:41405
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:42075
--webserver_port=38563
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:10.409226 8381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:10.409513 8381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:10.409679 8381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:10.417133 8381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:10.417258 8381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:10.417443 8381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:10.428133 8381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:41405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=38563
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:10.429725 8381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:10.431111 8381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:10.444818 8386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:10.445144 8387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:10.445250 8389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:10.445773 8381 server_base.cc:1034] running on GCE node
I20260927 22:41:10.446370 8381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:10.447315 8381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:10.448555 8381 hybrid_clock.cc:648] HybridClock initialized: now 1790548870448518 us; error 67 us; skew 500 ppm
I20260927 22:41:10.448876 8381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:10 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548870, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:10.452215 8381 init.cc:377] Logged in from keytab as oryx/127.1.53.66@KRBTEST.COM (short username oryx)
I20260927 22:41:10.453789 8381 webserver.cc:460] Webserver started at http://127.1.53.66:38563/ using document root <none> and password file <none>
I20260927 22:41:10.454521 8381 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:10.454682 8381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:10.460528 8381 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:41:10.463510 8396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:10.465103 8381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:10.465293 8381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "77fc447e03a044c0b06dad1b4ed79a75"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "ce7dd5334242b3668382ca43165cb565"
server_key_iv: "801ecc69cc93a7b2b47dc229d60ee9f3"
server_key_version: "encryptionkey@0"
I20260927 22:41:10.466382 8381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:10.483835 8381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:10.488557 8381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:10.488837 8381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:10.489861 8381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:10.491225 8381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:10.491348 8381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:10.491480 8381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:10.491573 8381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:10.526539 8381 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:41405
I20260927 22:41:10.526587 8508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:41405 every 8 connection(s)
I20260927 22:41:10.527997 8381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:10.537184 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 8381
I20260927 22:41:10.537545 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7716
Sep 27 22:41:10 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548870, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
Sep 27 22:41:10 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548870, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
Sep 27 22:41:10 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548870, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.124@KRBTEST.COM
Sep 27 22:41:10 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.67@KRBTEST.COM
I20260927 22:41:10.577972 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.530937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:52243 (local address 127.1.53.125:37173)
0927 22:41:10.551369 (+ 20432us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:10.551378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:10.551403 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:41:10.551481 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:10.551487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:10.551592 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:10.551727 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:10.554771 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.556215 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.559491 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.560158 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.560295 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.568392 (+ 8097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:10.568401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:10.568408 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:10.568453 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:10.573941 (+ 5488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.574939 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.574946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.574950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.575107 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.575677 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.575687 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.575690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.576067 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:10.576188 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:10.577281 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0927 22:41:10.577503 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20256,"thread_start_us":20913,"threads_started":1}
I20260927 22:41:10.579407 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.531924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:36717 (local address 127.1.53.124:36011)
0927 22:41:10.557589 (+ 25665us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:10.557626 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:10.559175 (+ 1549us) server_negotiation.cc:407] Connection header received
0927 22:41:10.559366 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:10.559373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:10.559470 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:10.559581 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:10.561321 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.563078 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.566861 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.567937 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.568035 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.572445 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:10.572452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:10.572457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:10.572500 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:10.576599 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.577356 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.577362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.577365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.577510 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.577889 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.577894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.577897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.578225 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:10.578345 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:10.578835 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 22:41:10.579044 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25342,"thread_start_us":25674,"threads_started":1}
I20260927 22:41:10.579671 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.532196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:51051)
0927 22:41:10.556462 (+ 24266us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:10.556489 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:10.556654 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:10.560163 (+ 3509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:10.560168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:10.560898 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:10.561284 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:10.561296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.562888 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.562894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:10.565244 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:10.565259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.566450 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.566456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:10.566706 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.567362 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:10.567390 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:10.570623 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:10.575702 (+ 5079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:10.575709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:10.575712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:10.576087 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:10.576503 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:10.576509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:10.576512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:10.576644 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:10.577280 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:10.577286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:10.577423 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 22:41:10.577564 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:41:10.578137 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24093,"mutex_wait_us":32,"thread_start_us":24513,"threads_started":1}
I20260927 22:41:10.579697 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.531735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.1.53.66:52243)
0927 22:41:10.542441 (+ 10706us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:10.542508 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:10.547029 (+ 4521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:10.552416 (+ 5387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:10.552434 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:10.553440 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:10.554501 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:10.554530 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.556474 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.556483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:10.559302 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:10.559318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.560690 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.560705 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:10.561064 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.564080 (+ 3016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:10.564115 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:10.568021 (+ 3906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:10.574113 (+ 6092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:10.574122 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:10.574142 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:10.574773 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:10.575267 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:10.575272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:10.575276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:10.575541 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:10.576319 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:10.576328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:10.576828 (+ 500us) client_negotiation.cc:769] Sending connection context
0927 22:41:10.578245 (+ 1417us) client_negotiation.cc:240] Negotiation successful
0927 22:41:10.578544 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10360,"mutex_wait_us":20,"thread_start_us":26889,"threads_started":1}
I20260927 22:41:10.580724 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.532439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.1.53.66:36717)
0927 22:41:10.559041 (+ 26602us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:10.559064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:10.559225 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:10.559779 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:10.559784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:10.560620 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:10.561112 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:10.561124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.563343 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.563347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:10.566694 (+ 3347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:10.566710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.568453 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.568459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:10.568676 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.569418 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:10.569446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:10.572169 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:10.576722 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:10.576728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:10.576731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:10.577238 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:10.577642 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:10.577646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:10.577649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:10.577772 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:10.578429 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:10.578433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:10.578571 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 22:41:10.579664 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0927 22:41:10.580333 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26451,"mutex_wait_us":67,"thread_start_us":9525,"threads_started":1}
I20260927 22:41:10.580842 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.531141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:51051 (local address 127.1.53.126:42075)
0927 22:41:10.559694 (+ 28553us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:10.559701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:10.559720 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:41:10.559775 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:10.559779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:10.559872 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:10.559989 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:10.561479 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.562643 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.565468 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.566232 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.567120 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.570872 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:10.570879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:10.570883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:10.570923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:10.575593 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.576217 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.576222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.576226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.576390 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.576801 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.576805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.576809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.577141 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:10.580100 (+ 2959us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:10.580260 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 22:41:10.580477 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28429,"thread_start_us":32045,"threads_started":1}
I20260927 22:41:10.581288 8511 heartbeater.cc:347] Connected to a master server at 127.1.53.125:37173
I20260927 22:41:10.581573 8509 heartbeater.cc:347] Connected to a master server at 127.1.53.124:36011
I20260927 22:41:10.581652 8511 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:10.581660 8510 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42075
I20260927 22:41:10.581694 8509 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:10.581820 8510 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:10.582741 8509 heartbeater.cc:511] Master 127.1.53.124:36011 requested a full tablet report, sending...
I20260927 22:41:10.582770 8510 heartbeater.cc:511] Master 127.1.53.126:42075 requested a full tablet report, sending...
I20260927 22:41:10.583595 8511 heartbeater.cc:511] Master 127.1.53.125:37173 requested a full tablet report, sending...
I20260927 22:41:10.584347 7387 ts_manager.cc:194] Re-registered known tserver with Master: 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:10.584631 8184 ts_manager.cc:194] Registered new tserver with Master: 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:10.584714 7988 ts_manager.cc:194] Registered new tserver with Master: 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:10.585394 7387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:51051
I20260927 22:41:10.585880 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:10.538598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.1.53.67:32915)
0927 22:41:10.558128 (+ 19530us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:10.558138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:10.560354 (+ 2216us) server_negotiation.cc:407] Connection header received
0927 22:41:10.560438 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:10.560459 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:10.560753 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:10.560923 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:10.563360 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.565719 (+ 2359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.567654 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:10.568613 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:10.568733 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:10.574649 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:10.574663 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:10.574682 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:10.574728 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:10.580344 (+ 5616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.582463 (+ 2119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.582473 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.582478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.582643 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:10.583097 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:10.583102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:10.583105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:10.583517 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:10.583843 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:10.584615 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 22:41:10.585416 (+ 801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19345,"thread_start_us":19541,"threads_started":1}
I20260927 22:41:10.587891 7839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:10.670523 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7716
I20260927 22:41:10.686721 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:32915
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:42075
--webserver_port=39307
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:41:10.961213 8370 heartbeater.cc:503] Master 127.1.53.126:42075 was elected leader, sending a full tablet report...
W20260927 22:41:11.026352 8522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:11.026691 8522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:11.026868 8522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:11.034286 8522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:11.034444 8522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:11.034628 8522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:11.045965 8522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:32915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=39307
--tserver_master_addrs=127.1.53.126:42075,127.1.53.125:37173,127.1.53.124:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:11.047477 8522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:11.048941 8522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:11.062903 8527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:11.063015 8530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:11.062939 8528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:11.065976 8522 server_base.cc:1034] running on GCE node
I20260927 22:41:11.066650 8522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:11.067649 8522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:11.068934 8522 hybrid_clock.cc:648] HybridClock initialized: now 1790548871068846 us; error 40 us; skew 500 ppm
I20260927 22:41:11.069311 8522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:11 dist-test-slave-pp80 krb5kdc[7342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548871, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:11.072674 8522 init.cc:377] Logged in from keytab as oryx/127.1.53.67@KRBTEST.COM (short username oryx)
I20260927 22:41:11.074222 8522 webserver.cc:460] Webserver started at http://127.1.53.67:39307/ using document root <none> and password file <none>
I20260927 22:41:11.075038 8522 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:11.075193 8522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:11.080679 8522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:11.083627 8537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:11.085079 8522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:11.085259 8522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "3eda5af2fa284054aaf632a0ffb4c870"
format_stamp: "Formatted at 2026-09-27 22:41:07 on dist-test-slave-pp80"
server_key: "577163846cce90a834978ff3b19fe0a3"
server_key_iv: "89ce0a5f5f5d4aea3fbb1f261c498612"
server_key_version: "encryptionkey@0"
I20260927 22:41:11.086282 8522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:11.126194 8522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:11.130872 8522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:11.131196 8522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:11.132231 8522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:11.133708 8522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:11.133831 8522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:11.133981 8522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:11.134070 8522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:11.167896 8522 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:32915
I20260927 22:41:11.167997 8649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:32915 every 8 connection(s)
I20260927 22:41:11.169479 8522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:11.179368 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 8522
Sep 27 22:41:11 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548871, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.124@KRBTEST.COM
Sep 27 22:41:11 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548871, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.125@KRBTEST.COM
Sep 27 22:41:11 dist-test-slave-pp80 krb5kdc[7342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548871, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:11.227031 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.172998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:49959 (local address 127.1.53.124:36011)
0927 22:41:11.189995 (+ 16997us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.190001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.190024 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:41:11.191227 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.191234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.191316 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.191408 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.194420 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.195809 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.198953 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.199875 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.199991 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.207106 (+ 7115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.207115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.207123 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.207169 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.213963 (+ 6794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.216558 (+ 2595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.216568 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.216574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.216775 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.217495 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.217503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.217509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.217983 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.218158 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.226430 (+ 8272us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.226666 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16879,"thread_start_us":17032,"threads_started":1}
I20260927 22:41:11.227052 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.172980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:44345 (local address 127.1.53.126:42075)
0927 22:41:11.202392 (+ 29412us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.202397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.202413 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:41:11.202459 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.202462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.202517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.202648 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.203930 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.205007 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.208518 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.209898 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.210027 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.215103 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.215113 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.215120 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.215176 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.222151 (+ 6975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.223778 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.223787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.223793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.224029 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.224590 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.224598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.224603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.225502 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.225673 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.226211 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.226506 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29305,"thread_start_us":29213,"threads_started":1}
I20260927 22:41:11.228143 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.173634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:44345)
0927 22:41:11.183719 (+ 10085us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.183777 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.191093 (+ 7316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.202672 (+ 11579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.202677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.203153 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:11.203556 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.203569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.205175 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.205182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.207310 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.207329 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.210101 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.210107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.210325 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.210941 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:11.210970 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:11.214873 (+ 3903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:11.222359 (+ 7486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:11.222368 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:11.222374 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:11.223443 (+ 1069us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:11.224147 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:11.224155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:11.224161 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:11.224393 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:11.225641 (+ 1248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:11.225650 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:11.225856 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.226046 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.226667 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9579,"mutex_wait_us":396,"thread_start_us":9324,"threads_started":1}
I20260927 22:41:11.229578 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.174253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.1.53.67:49959)
0927 22:41:11.184652 (+ 10399us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.184670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.190697 (+ 6027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.191861 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.191883 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.192998 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:11.194113 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.194147 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.196313 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.196319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.198703 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.198725 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.200412 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.200421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.200749 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.201685 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:11.201732 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:11.206793 (+ 5061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:11.214315 (+ 7522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:11.214327 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:11.214349 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:11.216381 (+ 2032us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:11.216967 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:11.216975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:11.216981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:11.217340 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:11.218289 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:11.218302 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:11.225232 (+ 6930us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.227744 (+ 2512us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.228031 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10273,"thread_start_us":25722,"threads_started":1}
I20260927 22:41:11.230211 8651 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42075
I20260927 22:41:11.230594 8651 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:11.231588 8651 heartbeater.cc:511] Master 127.1.53.126:42075 requested a full tablet report, sending...
I20260927 22:41:11.232007 8650 heartbeater.cc:347] Connected to a master server at 127.1.53.124:36011
I20260927 22:41:11.232153 8650 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:11.232609 8650 heartbeater.cc:511] Master 127.1.53.124:36011 requested a full tablet report, sending...
I20260927 22:41:11.233376 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.172693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:45987 (local address 127.1.53.125:37173)
0927 22:41:11.186609 (+ 13916us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.186615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.200509 (+ 13894us) server_negotiation.cc:407] Connection header received
0927 22:41:11.200650 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.200659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.200776 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.200927 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.204511 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.205789 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.208131 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.209077 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.209205 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.214230 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.214241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.214249 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.214306 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.222603 (+ 8297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.224207 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.224215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.224221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.224439 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.227847 (+ 3408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.227855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.227862 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.228294 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.228439 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.232787 (+ 4348us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.233028 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13782,"thread_start_us":13989,"threads_started":1}
I20260927 22:41:11.233412 7387 ts_manager.cc:194] Re-registered known tserver with Master: 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915)
I20260927 22:41:11.233744 8184 ts_manager.cc:194] Registered new tserver with Master: 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915)
I20260927 22:41:11.234495 7387 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:44345
I20260927 22:41:11.234692 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.174848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.1.53.67:45987)
0927 22:41:11.200366 (+ 25518us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.200387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.200516 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.201147 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.201155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.202145 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:11.202548 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.202565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.205996 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.206001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.207917 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.207938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.209579 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.209588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.209852 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.210560 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:11.210584 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:11.213917 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:11.222857 (+ 8940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:11.222864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:11.222871 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:11.223631 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:11.225765 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:11.225772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:11.225779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:11.225966 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:11.229035 (+ 3069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:11.229042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:11.229207 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.229368 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.234271 (+ 4903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25335,"mutex_wait_us":4,"thread_start_us":9562,"threads_started":1}
I20260927 22:41:11.239990 8652 heartbeater.cc:347] Connected to a master server at 127.1.53.125:37173
I20260927 22:41:11.240159 8652 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:11.240556 8652 heartbeater.cc:511] Master 127.1.53.125:37173 requested a full tablet report, sending...
I20260927 22:41:11.241664 7988 ts_manager.cc:194] Registered new tserver with Master: 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915)
I20260927 22:41:11.255470 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.206952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.1.53.126:42075)
0927 22:41:11.227341 (+ 20389us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.227348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.227369 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:41:11.227426 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.227434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.227526 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.227645 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.239663 (+ 12018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.240954 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.242874 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.243411 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.243530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.247239 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.247247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.247251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.247292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.251637 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.252478 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.252484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.252488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.252625 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.253082 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.253088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.253092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.253432 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.253711 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.254700 (+ 989us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.254980 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20252,"thread_start_us":23775,"threads_started":1}
I20260927 22:41:11.255659 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.211223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.1.53.125:37173)
0927 22:41:11.233692 (+ 22469us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.233697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.233714 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:41:11.233759 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.233764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.233828 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.233926 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.237655 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.241477 (+ 3822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.243601 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.244310 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.244438 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.247319 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.247326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.247330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.247365 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.251636 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.253000 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.253005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.253011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.253202 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.253734 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.253739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.253744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.254055 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.254321 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.255001 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.255234 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22366,"thread_start_us":23979,"threads_started":1}
I20260927 22:41:11.257491 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.218945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.1.53.124:36011)
0927 22:41:11.227683 (+ 8738us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.227689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.240286 (+ 12597us) server_negotiation.cc:407] Connection header received
0927 22:41:11.240544 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.240550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.240644 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.240718 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.243253 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.244208 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.246472 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.247160 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.247295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.250600 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.250606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.250609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.250644 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.253713 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.254512 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.254518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.254521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.254845 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.255476 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.255482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.255486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.255873 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.256121 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.256835 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.257108 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8643,"thread_start_us":11704,"threads_started":1}
I20260927 22:41:11.260224 7387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:11.267591 7387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:11.299659 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.289761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:41405 (local address 127.0.0.1:58370)
0927 22:41:11.290263 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.290280 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.290429 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.292017 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.292022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.292070 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:11.292378 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.292387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.294499 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.294507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.297003 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.297018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.298839 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.298844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.299006 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.299039 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.299140 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.299295 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
I20260927 22:41:11.299675 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.289804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.1.53.66:41405)
0927 22:41:11.291193 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.291200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.291224 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:41:11.291311 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.291316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.291592 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.291802 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:11.292522 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.294366 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.297184 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.298637 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.298775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.298980 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.299206 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.299389 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":1154,"threads_started":1}
I20260927 22:41:11.301945 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.288970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:32915 (local address 127.0.0.1:36914)
0927 22:41:11.289514 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.289532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.289638 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.294024 (+ 4386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.294032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.294068 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:11.295146 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.295165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.297241 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.297247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.299285 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.299298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.301113 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.301119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.301322 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.301362 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.301456 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.301586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":271,"threads_started":1}
I20260927 22:41:11.301985 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.289282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.1.53.67:32915)
0927 22:41:11.293016 (+ 3734us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.293024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.293058 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:41:11.293144 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.293151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.293367 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.293996 (+ 629us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:11.295423 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.297051 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.299491 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.300918 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.301053 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.301253 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.301513 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.301664 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":3393,"threads_started":1}
I20260927 22:41:11.303938 8444 tablet_service.cc:1467] Processing CreateTablet for tablet cc7a491b3a6e42658af6f7cc4ee8f8de (DEFAULT_TABLE table=test-table [id=9486f6551642457fb552289e69b02f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:11.304875 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.287836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44621 (local address 127.0.0.1:46718)
0927 22:41:11.288523 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.288542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.288663 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.297230 (+ 8567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.297234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.297259 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:11.297623 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.297631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.298896 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.300025 (+ 1129us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.301729 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.301743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.303628 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.303634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.303800 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.303841 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.303941 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.304071 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":22,"thread_start_us":568,"threads_started":1}
I20260927 22:41:11.305516 8444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7a491b3a6e42658af6f7cc4ee8f8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:11.305920 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.288278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.1.53.65:44621)
0927 22:41:11.296679 (+ 8401us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.296687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.296715 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:41:11.296801 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.296808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.297060 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.297237 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:11.297795 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.298747 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.301905 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.303404 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.303545 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.303740 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.305360 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.305615 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8237,"thread_start_us":8124,"threads_started":1}
I20260927 22:41:11.306439 8585 tablet_service.cc:1467] Processing CreateTablet for tablet cc7a491b3a6e42658af6f7cc4ee8f8de (DEFAULT_TABLE table=test-table [id=9486f6551642457fb552289e69b02f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:11.307673 8585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7a491b3a6e42658af6f7cc4ee8f8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:11.309517 8304 tablet_service.cc:1467] Processing CreateTablet for tablet cc7a491b3a6e42658af6f7cc4ee8f8de (DEFAULT_TABLE table=test-table [id=9486f6551642457fb552289e69b02f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:11.310828 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc7a491b3a6e42658af6f7cc4ee8f8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:11.318679 8687 tablet_bootstrap.cc:492] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Bootstrap starting.
I20260927 22:41:11.321995 8687 tablet_bootstrap.cc:654] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:11.323253 8688 tablet_bootstrap.cc:492] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Bootstrap starting.
I20260927 22:41:11.323451 8687 log.cc:824] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:11.328519 8688 tablet_bootstrap.cc:654] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:11.330267 8688 log.cc:824] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:11.331892 8690 tablet_bootstrap.cc:492] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Bootstrap starting.
I20260927 22:41:11.333696 8687 tablet_bootstrap.cc:492] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: No bootstrap required, opened a new log
I20260927 22:41:11.334005 8687 ts_tablet_manager.cc:1397] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.013s
I20260927 22:41:11.334249 8688 tablet_bootstrap.cc:492] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: No bootstrap required, opened a new log
I20260927 22:41:11.334517 8688 ts_tablet_manager.cc:1397] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260927 22:41:11.335485 8690 tablet_bootstrap.cc:654] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:11.337384 8690 log.cc:824] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:11.342304 8687 raft_consensus.cc:348] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.342914 8687 raft_consensus.cc:374] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:11.343086 8687 raft_consensus.cc:729] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fc447e03a044c0b06dad1b4ed79a75, State: Initialized, Role: FOLLOWER
I20260927 22:41:11.343796 8687 consensus_queue.cc:260] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.345966 8690 tablet_bootstrap.cc:492] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: No bootstrap required, opened a new log
I20260927 22:41:11.346264 8690 ts_tablet_manager.cc:1397] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 22:41:11.347127 8688 raft_consensus.cc:348] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.347469 8687 ts_tablet_manager.cc:1428] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260927 22:41:11.347543 8688 raft_consensus.cc:374] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:11.347649 8688 raft_consensus.cc:729] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eda5af2fa284054aaf632a0ffb4c870, State: Initialized, Role: FOLLOWER
I20260927 22:41:11.347770 8510 heartbeater.cc:503] Master 127.1.53.126:42075 was elected leader, sending a full tablet report...
I20260927 22:41:11.348380 8688 consensus_queue.cc:260] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.352828 8688 ts_tablet_manager.cc:1428] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
I20260927 22:41:11.353210 8651 heartbeater.cc:503] Master 127.1.53.126:42075 was elected leader, sending a full tablet report...
I20260927 22:41:11.359130 8690 raft_consensus.cc:348] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.359578 8690 raft_consensus.cc:374] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:11.359696 8690 raft_consensus.cc:729] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded01630111646d1a7fa7d6cc9c12b7a, State: Initialized, Role: FOLLOWER
I20260927 22:41:11.360287 8690 consensus_queue.cc:260] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.362007 8690 ts_tablet_manager.cc:1428] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260927 22:41:11.398850 8372 tablet.cc:2369] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:41:11.428092 8653 tablet.cc:2369] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:11.517064 8695 raft_consensus.cc:482] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:11.517431 8695 raft_consensus.cc:504] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.519157 8695 leader_election.cc:288] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915), 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:11.527212 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.519668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:50803 (local address 127.1.53.67:32915)
0927 22:41:11.519885 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.519913 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.520283 (+ 370us) server_negotiation.cc:407] Connection header received
0927 22:41:11.520628 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.520635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.520755 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.520893 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:11.521494 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.522335 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.525338 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.526306 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.526465 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.526568 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.526843 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.526967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:41:11.527309 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.519462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:32915 (local address 127.1.53.65:50803)
0927 22:41:11.520149 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.520171 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.520340 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.520914 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.520919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.520949 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:11.521334 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.521345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.522510 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.522519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.525078 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.525093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.526524 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.526530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.526686 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.526709 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.526818 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.526960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":359,"threads_started":1}
I20260927 22:41:11.528405 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.520365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:41405 (local address 127.1.53.65:55625)
0927 22:41:11.521019 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:11.521041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:11.521193 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:11.521526 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:11.521531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:11.521552 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:11.521836 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.521844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.523472 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.523477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.525383 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:11.525397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.527426 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.527433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:11.527623 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.527654 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:41:11.527755 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:41:11.527895 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":422,"threads_started":1}
I20260927 22:41:11.528329 8605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7a491b3a6e42658af6f7cc4ee8f8de" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eda5af2fa284054aaf632a0ffb4c870" is_pre_election: true
I20260927 22:41:11.528817 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.520580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:55625 (local address 127.1.53.66:41405)
0927 22:41:11.520790 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.520796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.521128 (+ 332us) server_negotiation.cc:407] Connection header received
0927 22:41:11.521323 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.521328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.521414 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.521509 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:11.522002 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.523321 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.525541 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.527018 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.527170 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.527268 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.527792 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.527930 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:41:11.528998 8605 raft_consensus.cc:2454] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 0.
I20260927 22:41:11.529665 8258 leader_election.cc:302] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eda5af2fa284054aaf632a0ffb4c870, ded01630111646d1a7fa7d6cc9c12b7a; no voters:
I20260927 22:41:11.529771 8464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7a491b3a6e42658af6f7cc4ee8f8de" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fc447e03a044c0b06dad1b4ed79a75" is_pre_election: true
I20260927 22:41:11.530011 8695 raft_consensus.cc:2781] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:11.530128 8695 raft_consensus.cc:482] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:11.530187 8695 raft_consensus.cc:3037] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:11.530295 8464 raft_consensus.cc:2454] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 0.
I20260927 22:41:11.532616 8695 raft_consensus.cc:504] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.533272 8695 leader_election.cc:288] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 election: Requested vote from peers 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915), 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:11.533766 8605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7a491b3a6e42658af6f7cc4ee8f8de" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eda5af2fa284054aaf632a0ffb4c870"
I20260927 22:41:11.533900 8464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc7a491b3a6e42658af6f7cc4ee8f8de" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fc447e03a044c0b06dad1b4ed79a75"
I20260927 22:41:11.534122 8464 raft_consensus.cc:3037] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:11.534435 8605 raft_consensus.cc:3037] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:11.536747 8605 raft_consensus.cc:2454] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 1.
I20260927 22:41:11.537206 8258 leader_election.cc:302] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eda5af2fa284054aaf632a0ffb4c870, ded01630111646d1a7fa7d6cc9c12b7a; no voters:
I20260927 22:41:11.537539 8695 raft_consensus.cc:2781] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:11.537822 8464 raft_consensus.cc:2454] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 1.
I20260927 22:41:11.538126 8695 raft_consensus.cc:686] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 1 LEADER]: Becoming Leader. State: Replica: ded01630111646d1a7fa7d6cc9c12b7a, State: Running, Role: LEADER
I20260927 22:41:11.539055 8695 consensus_queue.cc:237] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:11.545017 7386 catalog_manager.cc:5495] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a reported cstate change: term changed from 0 to 1, leader changed from <none> to ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65). New cstate: current_term: 1 leader_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } health_report { overall_health: UNKNOWN } } }
W20260927 22:41:11.559381 8512 tablet.cc:2369] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:11.595643 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.589814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.1.53.65:44621)
0927 22:41:11.590029 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.590035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.590055 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:41:11.590319 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.590324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.590426 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.590622 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:11.590995 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.591785 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.593391 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.593840 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.593991 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.594494 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:11.595041 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:11.595216 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.595280 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.595360 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:41:11.603945 8464 raft_consensus.cc:1261] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Refusing update from remote peer ded01630111646d1a7fa7d6cc9c12b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:11.604063 8605 raft_consensus.cc:1261] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Refusing update from remote peer ded01630111646d1a7fa7d6cc9c12b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:11.605037 8698 consensus_queue.cc:1035] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:11.605469 8695 consensus_queue.cc:1035] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:11.639644 7387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:11.640407 7387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:11.648331 7387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 057aa11ad898437fa599b88a33ccb839: Sending DeleteTablet for 3 replicas of tablet cc7a491b3a6e42658af6f7cc4ee8f8de
I20260927 22:41:11.649510 8585 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7a491b3a6e42658af6f7cc4ee8f8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:11 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:36914
I20260927 22:41:11.649475 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7a491b3a6e42658af6f7cc4ee8f8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:11 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:46718
I20260927 22:41:11.649767 8444 tablet_service.cc:1514] Processing DeleteTablet for tablet cc7a491b3a6e42658af6f7cc4ee8f8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:11 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:58370
I20260927 22:41:11.661180 8713 tablet_replica.cc:330] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: stopping tablet replica
I20260927 22:41:11.661696 8713 raft_consensus.cc:2229] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:11.662683 8713 raft_consensus.cc:2258] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:11.665189 8713 ts_tablet_manager.cc:1905] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:11.666355 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7355
I20260927 22:41:11.670066 8711 tablet_replica.cc:330] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: stopping tablet replica
I20260927 22:41:11.670512 8711 raft_consensus.cc:2229] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:11.671064 8711 raft_consensus.cc:2258] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:11.672987 7417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:11.680018 8711 ts_tablet_manager.cc:1905] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:11.685377 8712 tablet_replica.cc:330] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: stopping tablet replica
I20260927 22:41:11.685868 8712 raft_consensus.cc:2229] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:11.686573 8712 raft_consensus.cc:2258] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:11.687498 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.654120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.1.53.125:37173)
0927 22:41:11.654300 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.654307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.663501 (+ 9194us) server_negotiation.cc:407] Connection header received
0927 22:41:11.664656 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.664666 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.664758 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.664836 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.669915 (+ 5079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.671236 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.674660 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.675512 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.675631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.679464 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.679470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.679476 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.679526 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.683720 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.684603 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.684609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.684613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.684752 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.685198 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.685203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.685206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.685689 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.686077 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.686615 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.687150 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:41:11.694517 8713 ts_tablet_manager.cc:1918] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:11.694697 8713 log.cc:1196] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/cc7a491b3a6e42658af6f7cc4ee8f8de
I20260927 22:41:11.695274 8713 ts_tablet_manager.cc:1939] T cc7a491b3a6e42658af6f7cc4ee8f8de P 77fc447e03a044c0b06dad1b4ed79a75: Deleting consensus metadata
I20260927 22:41:11.699237 8712 ts_tablet_manager.cc:1905] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:11.703783 8711 ts_tablet_manager.cc:1918] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:11.703989 8711 log.cc:1196] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/cc7a491b3a6e42658af6f7cc4ee8f8de
I20260927 22:41:11.704596 8711 ts_tablet_manager.cc:1939] T cc7a491b3a6e42658af6f7cc4ee8f8de P 3eda5af2fa284054aaf632a0ffb4c870: Deleting consensus metadata
I20260927 22:41:11.705922 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.665204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.1.53.124:36011)
0927 22:41:11.665392 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.665398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.689988 (+ 24590us) server_negotiation.cc:407] Connection header received
0927 22:41:11.690185 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.690191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.690282 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.690403 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.692156 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.693334 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.695418 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.696082 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.696184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.699725 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.699731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.699736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.699777 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.702682 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.703337 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.703344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.703349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.703540 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.704067 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.704073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.704076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.704478 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.704701 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.705307 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.705558 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:41:11.713289 8712 ts_tablet_manager.cc:1918] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:11.713501 8712 log.cc:1196] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/cc7a491b3a6e42658af6f7cc4ee8f8de
I20260927 22:41:11.714140 8712 ts_tablet_manager.cc:1939] T cc7a491b3a6e42658af6f7cc4ee8f8de P ded01630111646d1a7fa7d6cc9c12b7a: Deleting consensus metadata
I20260927 22:41:11.821864 7372 catalog_manager.cc:4841] TS 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915): tablet cc7a491b3a6e42658af6f7cc4ee8f8de (table test-table [id=9486f6551642457fb552289e69b02f70]) successfully deleted
I20260927 22:41:11.822955 7373 catalog_manager.cc:4841] TS ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65:44621): tablet cc7a491b3a6e42658af6f7cc4ee8f8de (table test-table [id=9486f6551642457fb552289e69b02f70]) successfully deleted
I20260927 22:41:11.823829 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7355
I20260927 22:41:11.876175 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.851138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.1.53.124:36011)
0927 22:41:11.851369 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.851377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.858653 (+ 7276us) server_negotiation.cc:407] Connection header received
0927 22:41:11.859191 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.859198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.859285 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.859373 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.861677 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.862847 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.864975 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.865641 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.865739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.869621 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.869629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.869635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.869689 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.873220 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.874050 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.874056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.874059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.874217 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.874655 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.874660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.874663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.875012 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.875299 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.875625 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.875837 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:41:11.877708 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:11.850623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.1.53.125:37173)
0927 22:41:11.850820 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:11.850827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:11.862983 (+ 12156us) server_negotiation.cc:407] Connection header received
0927 22:41:11.863060 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:11.863064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:11.863131 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:11.863245 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:11.865155 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.865869 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.867443 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:11.867836 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:11.867979 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:11.871447 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:11.871453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:11.871456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:11.871488 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:11.874872 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.875654 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.875659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.875662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.875846 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:11.876311 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:11.876316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:11.876318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:11.876647 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:11.876913 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:11.877190 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:41:11.877394 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:41:12.626762 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:12.626005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:38683)
0927 22:41:12.626453 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:12.626577 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
W20260927 22:41:12.627264 8370 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:42075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:12.633052 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:12.632302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:47773)
0927 22:41:12.632767 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:12.632882 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
W20260927 22:41:12.633446 8651 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:42075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:12.634975 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:12.634198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:55751)
0927 22:41:12.634668 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:12.634796 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
W20260927 22:41:12.635558 8510 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:42075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:13.148451 8726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 057aa11ad898437fa599b88a33ccb839)
I20260927 22:41:13.148533 8727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 057aa11ad898437fa599b88a33ccb839)
I20260927 22:41:13.148736 8726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:13.148813 8727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:13.150213 8726 leader_election.cc:288] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075), 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125:37173)
I20260927 22:41:13.150619 8727 leader_election.cc:288] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075), 7548b756336348acbf16385a39e73add (127.1.53.124:36011)
I20260927 22:41:13.151857 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.150298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48884)
0927 22:41:13.151297 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.151523 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":683,"threads_started":1}
W20260927 22:41:13.152204 8170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:13.153292 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.150296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48880)
0927 22:41:13.152778 (+ 2482us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.152995 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2311,"threads_started":1}
I20260927 22:41:13.153982 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.153591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48894)
0927 22:41:13.153766 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.153840 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:41:13.154199 8170 leader_election.cc:334] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
W20260927 22:41:13.154191 7974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:13.155833 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.155424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48900)
0927 22:41:13.155595 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.155682 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:41:13.156066 7974 leader_election.cc:334] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:13.160863 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.151256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.1.53.124:36011)
0927 22:41:13.152261 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:13.152267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:13.153534 (+ 1267us) server_negotiation.cc:407] Connection header received
0927 22:41:13.153807 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:13.153812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:13.153879 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:13.153946 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:13.154887 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.156005 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.158634 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.159885 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.160008 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:13.160101 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:13.160400 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:41:13.160555 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":640,"threads_started":1}
I20260927 22:41:13.160931 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.150298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36011 (local address 127.0.0.1:51794)
0927 22:41:13.153438 (+ 3140us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.153457 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:13.153651 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:13.154243 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:13.154251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:13.154283 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:13.154644 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:13.154662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.156137 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.156148 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:13.158458 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:13.158472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.160044 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.160049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:13.160178 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:13.160205 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:41:13.160374 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 22:41:13.160508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3030,"mutex_wait_us":82,"thread_start_us":2068,"threads_started":1}
I20260927 22:41:13.161310 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.150947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:37173 (local address 127.0.0.1:60254)
0927 22:41:13.151510 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.151540 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:13.151679 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:13.154591 (+ 2912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:13.154599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:13.154636 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:13.155015 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:13.155025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.156764 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.156770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:13.158763 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:13.158780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.160467 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.160473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:13.160680 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:13.160720 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:41:13.160822 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:41:13.160974 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":310,"threads_started":1}
I20260927 22:41:13.161548 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.153403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.1.53.125:37173)
0927 22:41:13.154200 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:13.154207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:13.154228 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:41:13.154294 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:13.154298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:13.154392 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:13.155166 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:13.155485 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.156640 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.159014 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:13.160230 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:13.160395 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:13.160499 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:13.160860 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 22:41:13.161242 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"spinlock_wait_cycles":2304,"thread_start_us":534,"threads_started":1}
I20260927 22:41:13.161628 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f90e4b26e334500ba8d3b211087d65c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7548b756336348acbf16385a39e73add" is_pre_election: true
I20260927 22:41:13.162051 8194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f90e4b26e334500ba8d3b211087d65c in term 1.
I20260927 22:41:13.162060 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7548b756336348acbf16385a39e73add" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f90e4b26e334500ba8d3b211087d65c" is_pre_election: true
I20260927 22:41:13.162343 7998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7548b756336348acbf16385a39e73add in term 1.
I20260927 22:41:13.162566 7973 leader_election.cc:302] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f90e4b26e334500ba8d3b211087d65c, 7548b756336348acbf16385a39e73add; no voters: 057aa11ad898437fa599b88a33ccb839
I20260927 22:41:13.162840 8171 leader_election.cc:302] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f90e4b26e334500ba8d3b211087d65c, 7548b756336348acbf16385a39e73add; no voters: 057aa11ad898437fa599b88a33ccb839
I20260927 22:41:13.162978 8727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:41:13.163089 8727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 057aa11ad898437fa599b88a33ccb839)
I20260927 22:41:13.163187 8727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:41:13.163348 8726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:41:13.163473 8726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Starting leader election (detected failure of leader 057aa11ad898437fa599b88a33ccb839)
I20260927 22:41:13.163592 8726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:41:13.165511 8727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:13.165760 8726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:13.166234 8727 leader_election.cc:288] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [CANDIDATE]: Term 2 election: Requested vote from peers 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075), 7548b756336348acbf16385a39e73add (127.1.53.124:36011)
I20260927 22:41:13.166524 8726 leader_election.cc:288] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 2 election: Requested vote from peers 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075), 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125:37173)
I20260927 22:41:13.166817 8194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f90e4b26e334500ba8d3b211087d65c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7548b756336348acbf16385a39e73add"
I20260927 22:41:13.167004 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.166586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48932)
0927 22:41:13.166757 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.166855 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 22:41:13.166965 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7548b756336348acbf16385a39e73add" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f90e4b26e334500ba8d3b211087d65c"
I20260927 22:41:13.167138 8194 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f90e4b26e334500ba8d3b211087d65c in current term 2: Already voted for candidate 7548b756336348acbf16385a39e73add in this term.
I20260927 22:41:13.167238 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.166352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48920)
0927 22:41:13.166979 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.167071 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
I20260927 22:41:13.167240 7998 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7548b756336348acbf16385a39e73add in current term 2: Already voted for candidate 2f90e4b26e334500ba8d3b211087d65c in this term.
I20260927 22:41:13.168514 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.168087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48940)
0927 22:41:13.168283 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.168368 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 22:41:13.168614 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.168176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:48956)
0927 22:41:13.168372 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.168454 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 22:41:13.168709 8170 leader_election.cc:334] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
W20260927 22:41:13.168751 7974 leader_election.cc:334] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:13.168830 7974 leader_election.cc:302] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f90e4b26e334500ba8d3b211087d65c; no voters: 057aa11ad898437fa599b88a33ccb839, 7548b756336348acbf16385a39e73add
I20260927 22:41:13.168823 8170 leader_election.cc:302] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7548b756336348acbf16385a39e73add; no voters: 057aa11ad898437fa599b88a33ccb839, 2f90e4b26e334500ba8d3b211087d65c
I20260927 22:41:13.169117 8726 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 22:41:13.169341 8727 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 22:41:13.628856 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.628111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:57555)
0927 22:41:13.628586 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.628672 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":269,"threads_started":1}
I20260927 22:41:13.634904 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.634153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:35781)
0927 22:41:13.634650 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.634740 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":286,"threads_started":1}
I20260927 22:41:13.637302 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:13.636208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:53119)
0927 22:41:13.636976 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:13.637078 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":295,"threads_started":1}
I20260927 22:41:14.630596 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.629832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:58469)
0927 22:41:14.630328 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.630425 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":253,"threads_started":1}
I20260927 22:41:14.636543 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.635735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:38433)
0927 22:41:14.636306 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.636384 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":328,"threads_started":1}
I20260927 22:41:14.640429 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.639162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:45173)
0927 22:41:14.640115 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.640189 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":796,"threads_started":1}
I20260927 22:41:14.675395 8742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:14.675625 8742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:14.676345 8742 leader_election.cc:288] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075), 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125:37173)
I20260927 22:41:14.676888 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7548b756336348acbf16385a39e73add" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f90e4b26e334500ba8d3b211087d65c" is_pre_election: true
I20260927 22:41:14.677142 7998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7548b756336348acbf16385a39e73add in term 2.
I20260927 22:41:14.677565 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.676617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49072)
0927 22:41:14.677196 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.677339 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":330,"threads_started":1}
I20260927 22:41:14.677605 8171 leader_election.cc:302] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f90e4b26e334500ba8d3b211087d65c, 7548b756336348acbf16385a39e73add; no voters:
I20260927 22:41:14.677897 8742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 22:41:14.678033 8742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:14.678150 8742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:41:14.679085 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.678694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49076)
0927 22:41:14.678861 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.678944 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:41:14.679270 8170 leader_election.cc:334] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:14.680637 8742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:14.681181 8742 leader_election.cc:288] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 3 election: Requested vote from peers 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075), 2f90e4b26e334500ba8d3b211087d65c (127.1.53.125:37173)
I20260927 22:41:14.681715 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.681340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49090)
0927 22:41:14.681532 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.681598 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:41:14.681746 7998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7548b756336348acbf16385a39e73add" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f90e4b26e334500ba8d3b211087d65c"
I20260927 22:41:14.681964 7998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:41:14.682615 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.682280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49100)
0927 22:41:14.682419 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.682490 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:41:14.682768 8170 leader_election.cc:334] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
I20260927 22:41:14.684060 7998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7548b756336348acbf16385a39e73add in term 3.
I20260927 22:41:14.684499 8171 leader_election.cc:302] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f90e4b26e334500ba8d3b211087d65c, 7548b756336348acbf16385a39e73add; no voters: 057aa11ad898437fa599b88a33ccb839
I20260927 22:41:14.684787 8742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:41:14.685156 8742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [term 3 LEADER]: Becoming Leader. State: Replica: 7548b756336348acbf16385a39e73add, State: Running, Role: LEADER
I20260927 22:41:14.685575 8742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } }
I20260927 22:41:14.687168 8745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7548b756336348acbf16385a39e73add. Latest consensus state: current_term: 3 leader_uuid: "7548b756336348acbf16385a39e73add" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:14.687417 8745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:14.688179 8747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:14.691875 8747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:14.692677 8747 catalog_manager.cc:1259] Loaded cluster ID: cadfcac557c94fdeb02e19f5913049bd
I20260927 22:41:14.692744 8747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:14.693771 8747 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:14.694497 8747 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Loaded TSK: 0
I20260927 22:41:14.695300 8747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:41:14.794672 8184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:14.802317 8184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:14.806939 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.806376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49120)
0927 22:41:14.806643 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.806746 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:41:14.806936 7998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [term 3 FOLLOWER]: Refusing update from remote peer 7548b756336348acbf16385a39e73add: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 22:41:14.807732 8745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 22:41:14.808089 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.807655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49124)
0927 22:41:14.807855 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.807932 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:41:14.808336 8170 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add -> Peer 057aa11ad898437fa599b88a33ccb839 (127.1.53.126:42075): Couldn't send request to peer 057aa11ad898437fa599b88a33ccb839. Status: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:41:14.810842 8749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7548b756336348acbf16385a39e73add. Latest consensus state: current_term: 3 leader_uuid: "7548b756336348acbf16385a39e73add" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:14.811151 8749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:14.813071 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7548b756336348acbf16385a39e73add" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:14.813390 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:14.813275 8749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7548b756336348acbf16385a39e73add" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "057aa11ad898437fa599b88a33ccb839" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42075 } } peers { permanent_uuid: "2f90e4b26e334500ba8d3b211087d65c" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 37173 } attrs { promote: false } } peers { permanent_uuid: "7548b756336348acbf16385a39e73add" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36011 } attrs { promote: false } } }
I20260927 22:41:14.813544 8749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f90e4b26e334500ba8d3b211087d65c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:41:14.831003 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.822188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44621 (local address 127.0.0.1:54658)
0927 22:41:14.822329 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.822351 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:14.822469 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:14.823304 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:14.823309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:14.823335 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:14.823621 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:14.823629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.825550 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.825559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:14.828470 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:14.828492 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.830233 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.830239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:14.830419 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:14.830443 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:41:14.830548 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:41:14.830677 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 22:41:14.831050 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.822377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.1.53.65:44621)
0927 22:41:14.822978 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:14.822987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:14.823013 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:41:14.823103 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:14.823110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:14.823184 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:14.823289 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:14.823772 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.824772 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.829075 (+ 4303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.830064 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.830223 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:14.830324 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:14.830574 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:41:14.830703 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"spinlock_wait_cycles":9472,"thread_start_us":410,"threads_started":1}
I20260927 22:41:14.832078 8304 tablet_service.cc:1467] Processing CreateTablet for tablet 21191da59a4146698cbc9b18a0617124 (DEFAULT_TABLE table=test-table [id=3dcc0fcecd3a48f9b5a1e4d4df35c461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:14.832558 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21191da59a4146698cbc9b18a0617124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:14.836526 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.825151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.1.53.66:41405)
0927 22:41:14.829142 (+ 3991us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:14.829148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:14.829166 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:41:14.829229 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:14.829235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:14.829314 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:14.829421 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:14.831335 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.832189 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.834160 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.835365 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.835512 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:14.835636 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:14.835988 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 22:41:14.836142 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3871,"thread_start_us":3769,"threads_started":1}
I20260927 22:41:14.838665 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.824909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:41405 (local address 127.0.0.1:43440)
0927 22:41:14.825624 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.825650 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:14.825837 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:14.829445 (+ 3608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:14.829453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:14.829490 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:14.829937 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:14.829953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.832341 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.832345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:14.833997 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:14.834009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.835593 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.835600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:14.835789 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:14.835825 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:41:14.835931 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:41:14.836071 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":3691,"threads_started":1}
I20260927 22:41:14.838689 8760 tablet_bootstrap.cc:492] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Bootstrap starting.
I20260927 22:41:14.839731 8444 tablet_service.cc:1467] Processing CreateTablet for tablet 21191da59a4146698cbc9b18a0617124 (DEFAULT_TABLE table=test-table [id=3dcc0fcecd3a48f9b5a1e4d4df35c461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:14.840176 8444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21191da59a4146698cbc9b18a0617124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:14.840359 8760 tablet_bootstrap.cc:654] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:14.841334 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.822929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.1.53.67:32915)
0927 22:41:14.834268 (+ 11339us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:14.834273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:14.834289 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:41:14.834346 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:14.834350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:14.834416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:14.834551 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:14.835130 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.836223 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.839072 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.840291 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.840436 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:14.840542 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:14.840882 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 22:41:14.841026 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11232,"thread_start_us":9231,"threads_started":1}
I20260927 22:41:14.841399 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.822804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:32915 (local address 127.0.0.1:47724)
0927 22:41:14.823490 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.823515 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:14.823929 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:14.834563 (+ 10634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:14.834568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:14.834594 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:14.834936 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:14.834956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.836377 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.836386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:14.838313 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:14.838326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:14.840489 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:14.840496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:14.840682 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:14.840726 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:41:14.840837 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:41:14.841064 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
I20260927 22:41:14.845930 8762 tablet_bootstrap.cc:492] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Bootstrap starting.
I20260927 22:41:14.846448 8585 tablet_service.cc:1467] Processing CreateTablet for tablet 21191da59a4146698cbc9b18a0617124 (DEFAULT_TABLE table=test-table [id=3dcc0fcecd3a48f9b5a1e4d4df35c461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:14.846948 8585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21191da59a4146698cbc9b18a0617124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:14.847589 8760 tablet_bootstrap.cc:492] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: No bootstrap required, opened a new log
I20260927 22:41:14.847738 8760 ts_tablet_manager.cc:1397] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 22:41:14.848033 8762 tablet_bootstrap.cc:654] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:14.848573 8760 raft_consensus.cc:348] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:14.848829 8760 raft_consensus.cc:374] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:14.848948 8760 raft_consensus.cc:729] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded01630111646d1a7fa7d6cc9c12b7a, State: Initialized, Role: FOLLOWER
I20260927 22:41:14.849201 8760 consensus_queue.cc:260] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:14.851171 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.850550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:57201)
0927 22:41:14.850840 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.850970 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 22:41:14.853389 8760 ts_tablet_manager.cc:1428] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260927 22:41:14.853736 8369 heartbeater.cc:503] Master 127.1.53.124:36011 was elected leader, sending a full tablet report...
I20260927 22:41:14.855728 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.855107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:50047)
0927 22:41:14.855409 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.855519 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 22:41:14.856853 8765 tablet_bootstrap.cc:492] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Bootstrap starting.
I20260927 22:41:14.857659 8762 tablet_bootstrap.cc:492] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: No bootstrap required, opened a new log
I20260927 22:41:14.857832 8762 ts_tablet_manager.cc:1397] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 22:41:14.858501 8762 raft_consensus.cc:348] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:14.858711 8762 raft_consensus.cc:374] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:14.858780 8765 tablet_bootstrap.cc:654] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:14.858803 8762 raft_consensus.cc:729] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fc447e03a044c0b06dad1b4ed79a75, State: Initialized, Role: FOLLOWER
I20260927 22:41:14.859053 8762 consensus_queue.cc:260] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:14.859994 8762 ts_tablet_manager.cc:1428] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:41:14.862104 8765 tablet_bootstrap.cc:492] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: No bootstrap required, opened a new log
I20260927 22:41:14.862272 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.860748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:38059)
0927 22:41:14.861994 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.862100 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1136}
I20260927 22:41:14.862278 8765 ts_tablet_manager.cc:1397] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 22:41:14.862366 8509 heartbeater.cc:503] Master 127.1.53.124:36011 was elected leader, sending a full tablet report...
I20260927 22:41:14.863052 8765 raft_consensus.cc:348] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:14.863317 8765 raft_consensus.cc:374] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:14.863430 8765 raft_consensus.cc:729] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eda5af2fa284054aaf632a0ffb4c870, State: Initialized, Role: FOLLOWER
I20260927 22:41:14.863662 8765 consensus_queue.cc:260] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:14.864377 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.863924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:49095)
0927 22:41:14.864099 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.864195 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 22:41:14.864527 8765 ts_tablet_manager.cc:1428] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:41:14.865442 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.865057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:45015)
0927 22:41:14.865258 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.865324 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 22:41:14.866449 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:14.866095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:50281)
0927 22:41:14.866230 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:14.866303 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:41:14.867210 8650 heartbeater.cc:503] Master 127.1.53.124:36011 was elected leader, sending a full tablet report...
I20260927 22:41:15.001821 8764 raft_consensus.cc:482] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:15.002103 8764 raft_consensus.cc:504] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:15.002849 8764 leader_election.cc:288] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915), 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:15.003595 8605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21191da59a4146698cbc9b18a0617124" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eda5af2fa284054aaf632a0ffb4c870" is_pre_election: true
I20260927 22:41:15.003680 8464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21191da59a4146698cbc9b18a0617124" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fc447e03a044c0b06dad1b4ed79a75" is_pre_election: true
I20260927 22:41:15.003896 8605 raft_consensus.cc:2454] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 0.
I20260927 22:41:15.003922 8464 raft_consensus.cc:2454] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 0.
I20260927 22:41:15.004448 8258 leader_election.cc:302] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eda5af2fa284054aaf632a0ffb4c870, ded01630111646d1a7fa7d6cc9c12b7a; no voters:
I20260927 22:41:15.004732 8764 raft_consensus.cc:2781] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:15.004868 8764 raft_consensus.cc:482] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:15.005033 8764 raft_consensus.cc:3037] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:15.007434 8764 raft_consensus.cc:504] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:15.007990 8764 leader_election.cc:288] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 election: Requested vote from peers 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915), 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405)
I20260927 22:41:15.008611 8605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21191da59a4146698cbc9b18a0617124" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eda5af2fa284054aaf632a0ffb4c870"
I20260927 22:41:15.008679 8464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21191da59a4146698cbc9b18a0617124" candidate_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fc447e03a044c0b06dad1b4ed79a75"
I20260927 22:41:15.008818 8605 raft_consensus.cc:3037] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:15.008932 8464 raft_consensus.cc:3037] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:15.010938 8464 raft_consensus.cc:2454] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 1.
I20260927 22:41:15.011094 8605 raft_consensus.cc:2454] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded01630111646d1a7fa7d6cc9c12b7a in term 1.
I20260927 22:41:15.011371 8258 leader_election.cc:302] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77fc447e03a044c0b06dad1b4ed79a75, ded01630111646d1a7fa7d6cc9c12b7a; no voters:
I20260927 22:41:15.011694 8764 raft_consensus.cc:2781] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:15.012259 8764 raft_consensus.cc:686] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 1 LEADER]: Becoming Leader. State: Replica: ded01630111646d1a7fa7d6cc9c12b7a, State: Running, Role: LEADER
I20260927 22:41:15.012588 8764 consensus_queue.cc:237] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } }
I20260927 22:41:15.013535 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.013097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:52869)
0927 22:41:15.013256 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.013366 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:41:15.015990 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.015544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:40607)
0927 22:41:15.015728 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.015817 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:41:15.016047 8183 catalog_manager.cc:5495] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a reported cstate change: term changed from 0 to 1, leader changed from <none> to ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65). New cstate: current_term: 1 leader_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded01630111646d1a7fa7d6cc9c12b7a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20260927 22:41:15.074064 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.069460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.1.53.65:44621)
0927 22:41:15.069712 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:15.069720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:15.069745 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:41:15.069802 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:15.069806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:15.069866 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:15.069980 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:15.070342 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.071168 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.072441 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.072833 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.072969 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:15.073381 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:15.073475 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:15.073576 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:15.073725 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 22:41:15.073800 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:41:15.077083 8464 raft_consensus.cc:1261] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Refusing update from remote peer ded01630111646d1a7fa7d6cc9c12b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:15.077173 8605 raft_consensus.cc:1261] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Refusing update from remote peer ded01630111646d1a7fa7d6cc9c12b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:15.077760 8769 consensus_queue.cc:1035] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77fc447e03a044c0b06dad1b4ed79a75" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:15.078070 8764 consensus_queue.cc:1035] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eda5af2fa284054aaf632a0ffb4c870" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:15.080916 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.080133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:36943)
0927 22:41:15.080654 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.080750 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413}
I20260927 22:41:15.080914 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.080517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:47645)
0927 22:41:15.080681 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.080762 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:41:15.086979 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.086246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.67:57961)
0927 22:41:15.086603 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.086719 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 22:41:15.098091 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.097519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:48687)
0927 22:41:15.097885 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.097960 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260927 22:41:15.105227 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.104350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.66:46859)
0927 22:41:15.104764 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.104904 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260927 22:41:15.106444 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.106013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.1.53.65:45873)
0927 22:41:15.106178 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.106274 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:41:15.106626 8183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:15.106920 8183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:15.113971 8183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7548b756336348acbf16385a39e73add: Sending DeleteTablet for 3 replicas of tablet 21191da59a4146698cbc9b18a0617124
I20260927 22:41:15.114920 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet 21191da59a4146698cbc9b18a0617124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:15 UTC) from {username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:54658
I20260927 22:41:15.115087 8444 tablet_service.cc:1514] Processing DeleteTablet for tablet 21191da59a4146698cbc9b18a0617124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:15 UTC) from {username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:43440
I20260927 22:41:15.115124 8585 tablet_service.cc:1514] Processing DeleteTablet for tablet 21191da59a4146698cbc9b18a0617124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:15 UTC) from {username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:47724
I20260927 22:41:15.115455 8443 tablet_service.cc:1514] Processing DeleteTablet for tablet 21191da59a4146698cbc9b18a0617124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:15 UTC) from {username='oryx', principal='oryx/127.1.53.124@KRBTEST.COM'} at 127.0.0.1:43440
I20260927 22:41:15.115682 8781 tablet_replica.cc:330] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: stopping tablet replica
I20260927 22:41:15.115967 8781 raft_consensus.cc:2229] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:15.116482 8781 raft_consensus.cc:2258] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:15.118276 8781 ts_tablet_manager.cc:1905] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:15.118686 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 8159
I20260927 22:41:15.126958 8781 ts_tablet_manager.cc:1918] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:15.127139 8781 log.cc:1196] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/21191da59a4146698cbc9b18a0617124
I20260927 22:41:15.127647 8781 ts_tablet_manager.cc:1939] T 21191da59a4146698cbc9b18a0617124 P ded01630111646d1a7fa7d6cc9c12b7a: Deleting consensus metadata
I20260927 22:41:15.129221 8170 catalog_manager.cc:4841] TS ded01630111646d1a7fa7d6cc9c12b7a (127.1.53.65:44621): tablet 21191da59a4146698cbc9b18a0617124 (table test-table [id=3dcc0fcecd3a48f9b5a1e4d4df35c461]) successfully deleted
I20260927 22:41:15.129657 8783 tablet_replica.cc:330] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: stopping tablet replica
I20260927 22:41:15.129958 8783 raft_consensus.cc:2229] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:15.130338 8783 raft_consensus.cc:2258] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:15.131336 8783 ts_tablet_manager.cc:1905] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:15.132714 8782 tablet_replica.cc:330] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: stopping tablet replica
I20260927 22:41:15.132901 8782 raft_consensus.cc:2229] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:15.133249 8782 raft_consensus.cc:2258] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:15.134258 8782 ts_tablet_manager.cc:1905] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:15.137601 8783 ts_tablet_manager.cc:1918] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:15.137737 8783 log.cc:1196] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/21191da59a4146698cbc9b18a0617124
I20260927 22:41:15.138185 8783 ts_tablet_manager.cc:1939] T 21191da59a4146698cbc9b18a0617124 P 3eda5af2fa284054aaf632a0ffb4c870: Deleting consensus metadata
I20260927 22:41:15.139273 8169 catalog_manager.cc:4841] TS 3eda5af2fa284054aaf632a0ffb4c870 (127.1.53.67:32915): tablet 21191da59a4146698cbc9b18a0617124 (table test-table [id=3dcc0fcecd3a48f9b5a1e4d4df35c461]) successfully deleted
I20260927 22:41:15.140553 8782 ts_tablet_manager.cc:1918] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:15.140715 8782 log.cc:1196] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/21191da59a4146698cbc9b18a0617124
I20260927 22:41:15.141119 8782 ts_tablet_manager.cc:1939] T 21191da59a4146698cbc9b18a0617124 P 77fc447e03a044c0b06dad1b4ed79a75: Deleting consensus metadata
I20260927 22:41:15.141970 8169 catalog_manager.cc:4841] TS 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405): tablet 21191da59a4146698cbc9b18a0617124 (table test-table [id=3dcc0fcecd3a48f9b5a1e4d4df35c461]) successfully deleted
W20260927 22:41:15.142575 8169 catalog_manager.cc:4808] TS 77fc447e03a044c0b06dad1b4ed79a75 (127.1.53.66:41405): delete failed for tablet 21191da59a4146698cbc9b18a0617124 because the tablet was not found. No further retry: Not found: Tablet not found: 21191da59a4146698cbc9b18a0617124
I20260927 22:41:15.145205 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.120555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.1.53.65:44621)
0927 22:41:15.120984 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:15.120994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:15.127944 (+ 6950us) server_negotiation.cc:407] Connection header received
0927 22:41:15.128324 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:15.128333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:15.128495 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:15.128597 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:15.132987 (+ 4390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.133968 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.135516 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.135924 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.136106 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:15.138898 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:15.138919 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:15.138935 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:15.138968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:15.142491 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:15.143344 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:15.143351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:15.143354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:15.143471 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:15.143799 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:15.143802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:15.143804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:15.144141 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:15.144429 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:15.144671 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:41:15.144829 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260927 22:41:15.145843 8364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:15.228827 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.228374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49138)
0927 22:41:15.228577 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.228674 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:41:15.229923 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.229545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49152)
0927 22:41:15.229700 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.229792 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:41:15.266455 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 8159
I20260927 22:41:15.290655 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 8381
I20260927 22:41:15.306195 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.291630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.1.53.66:41405)
0927 22:41:15.291826 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:15.291835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:15.291879 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:41:15.292159 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:15.292164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:15.292232 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:15.292400 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:15.293853 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.294615 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.295948 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.296528 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.296672 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:15.299525 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:15.299541 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:15.299561 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:15.299600 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:15.302780 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:15.303601 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:15.303609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:15.303612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:15.303768 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:15.304208 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:15.304216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:15.304219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:15.304857 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:15.305174 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:15.305535 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 22:41:15.305787 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:41:15.306936 8504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:15.402258 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 8381
I20260927 22:41:15.420461 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 8522
I20260927 22:41:15.434435 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.421311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.1.53.67:32915)
0927 22:41:15.421809 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:15.421815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:15.421842 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:41:15.421899 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:15.421904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:15.421971 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:15.422094 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:15.423649 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.424515 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.425741 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:15.426221 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:15.426324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:15.428818 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:15.428831 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:15.428848 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:15.428881 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:15.431739 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:15.432447 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:15.432454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:15.432457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:15.432616 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:15.433074 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:15.433077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:15.433079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:15.433427 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:15.433666 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:15.433944 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:41:15.434153 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":287,"threads_started":1}
I20260927 22:41:15.435083 8645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:15.529191 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 8522
I20260927 22:41:15.546952 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7849
I20260927 22:41:15.548164 8018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:15.654537 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.654050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49166)
0927 22:41:15.654258 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.654365 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:41:15.655628 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:15.655279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42075 (local address 127.0.0.1:49174)
0927 22:41:15.655440 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:15.655513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:41:15.667057 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7849
I20260927 22:41:15.689819 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 7954
I20260927 22:41:15.690948 8214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:15.812440 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 7954
2026-09-27T22:41:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11381 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:15 dist-test-slave-pp80 krb5kdc[8793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:15 dist-test-slave-pp80 krb5kdc[8793](info): set up 2 sockets
Sep 27 22:41:15 dist-test-slave-pp80 krb5kdc[8793](info): commencing operation
krb5kdc: starting...
W20260927 22:41:16.909179 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:17 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab.
Entry for principal oryx/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab.
I20260927 22:41:17.137691 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:35005
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab
--principal=oryx/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:35005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:17.470235 8806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:17.470528 8806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:17.470683 8806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:17.478339 8806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:17.478456 8806 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:41:17.478534 8806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:17.478631 8806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:17.478724 8806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:17.478814 8806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:17.489143 8806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:35005
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.126.keytab
--principal=oryx/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:35005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:17.490483 8806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:17.491819 8806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:17.504573 8811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:17.504647 8814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:17.504515 8812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:17.504895 8806 server_base.cc:1034] running on GCE node
I20260927 22:41:17.505944 8806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:17.507398 8806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:17.508646 8806 hybrid_clock.cc:648] HybridClock initialized: now 1790548877508576 us; error 46 us; skew 500 ppm
I20260927 22:41:17.508985 8806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:17 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548877, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:17.519357 8806 init.cc:377] Logged in from keytab as oryx/127.1.53.126@KRBTEST.COM (short username oryx)
I20260927 22:41:17.520955 8806 webserver.cc:460] Webserver started at http://127.1.53.126:36953/ using document root <none> and password file <none>
I20260927 22:41:17.521719 8806 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:17.521811 8806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:17.522085 8806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:17.525286 8806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "1374a0ecdbd74477b19d6de93b73e842"
format_stamp: "Formatted at 2026-09-27 22:41:17 on dist-test-slave-pp80"
server_key: "5cbc5303dc5587b08640d3f52410206b"
server_key_iv: "e1093ff96fa741f2c369f0666d8f0fbf"
server_key_version: "encryptionkey@0"
I20260927 22:41:17.526084 8806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "1374a0ecdbd74477b19d6de93b73e842"
format_stamp: "Formatted at 2026-09-27 22:41:17 on dist-test-slave-pp80"
server_key: "5cbc5303dc5587b08640d3f52410206b"
server_key_iv: "e1093ff96fa741f2c369f0666d8f0fbf"
server_key_version: "encryptionkey@0"
I20260927 22:41:17.531361 8806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:41:17.535231 8821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:17.536695 8806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:41:17.536836 8806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "1374a0ecdbd74477b19d6de93b73e842"
format_stamp: "Formatted at 2026-09-27 22:41:17 on dist-test-slave-pp80"
server_key: "5cbc5303dc5587b08640d3f52410206b"
server_key_iv: "e1093ff96fa741f2c369f0666d8f0fbf"
server_key_version: "encryptionkey@0"
I20260927 22:41:17.537007 8806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:17.560087 8806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:17.564733 8806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:17.565032 8806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:17.592634 8806 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:35005
I20260927 22:41:17.592648 8882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:35005 every 8 connection(s)
I20260927 22:41:17.594013 8806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:17.598973 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:17.600229 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 8806
I20260927 22:41:17.600471 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:17.601303 1237 external_mini_cluster.cc:1428] Setting key 76967929f67fad9aac6af9df0e3a0a41
I20260927 22:41:17.612031 8883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Bootstrap starting.
I20260927 22:41:17.615989 8883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:17.617076 8883 log.cc:824] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:17.620349 8883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: No bootstrap required, opened a new log
Sep 27 22:41:17 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:17.630899 8883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1374a0ecdbd74477b19d6de93b73e842" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35005 } }
I20260927 22:41:17.631356 8883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:17.631461 8883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1374a0ecdbd74477b19d6de93b73e842, State: Initialized, Role: FOLLOWER
I20260927 22:41:17.631980 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:17.603776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.1.53.126:35005)
0927 22:41:17.604833 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:17.604850 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:17.608946 (+ 4096us) server_negotiation.cc:407] Connection header received
0927 22:41:17.611378 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:17.611409 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:17.611980 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:17.612508 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:17.613820 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:17.615794 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:17.617238 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:17.618198 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:17.618350 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:17.622790 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:17.622801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:17.622821 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:17.622871 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:17.626970 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:17.627696 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:17.627705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:17.627717 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:17.627928 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:17.628481 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:17.628487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:17.628490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:17.629264 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:17.629586 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:17.629994 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 22:41:17.630447 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":440,"threads_started":1}
I20260927 22:41:17.632140 8883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1374a0ecdbd74477b19d6de93b73e842" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35005 } }
I20260927 22:41:17.632464 8883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:17.632604 8883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:17.632795 8883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:17.635694 8883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1374a0ecdbd74477b19d6de93b73e842" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35005 } }
I20260927 22:41:17.636307 8883 leader_election.cc:302] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1374a0ecdbd74477b19d6de93b73e842; no voters:
I20260927 22:41:17.636941 8883 leader_election.cc:288] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:17.637106 8888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:17.638228 8888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [term 1 LEADER]: Becoming Leader. State: Replica: 1374a0ecdbd74477b19d6de93b73e842, State: Running, Role: LEADER
I20260927 22:41:17.638756 8888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1374a0ecdbd74477b19d6de93b73e842" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35005 } }
I20260927 22:41:17.639158 8883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:17.641525 8889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1374a0ecdbd74477b19d6de93b73e842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1374a0ecdbd74477b19d6de93b73e842" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35005 } } }
I20260927 22:41:17.641876 8889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:17.641619 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1374a0ecdbd74477b19d6de93b73e842. Latest consensus state: current_term: 1 leader_uuid: "1374a0ecdbd74477b19d6de93b73e842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1374a0ecdbd74477b19d6de93b73e842" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 35005 } } }
I20260927 22:41:17.641999 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:17.645026 8895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:17.649959 8895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:17.660754 8895 catalog_manager.cc:1347] Generated new cluster ID: a8f0affa52da4e01825f22b37c3e9a4d
I20260927 22:41:17.660929 8895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:17.678583 8895 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:17.680467 8895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:17.688519 8895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Generated new TSK 0
I20260927 22:41:17.689502 8895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab.
Entry for principal oryx/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab.
I20260927 22:41:17.764017 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35005
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:18.121469 8909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:18.121834 8909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:18.121939 8909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:18.129412 8909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:18.129541 8909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:18.129629 8909 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:41:18.129710 8909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:18.139715 8909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:35005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:18.141312 8909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:18.142688 8909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:18.156407 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:18.156445 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:18.156342 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:18.157363 8909 server_base.cc:1034] running on GCE node
I20260927 22:41:18.158007 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:18.158988 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:18.160295 8909 hybrid_clock.cc:648] HybridClock initialized: now 1790548878160188 us; error 41 us; skew 500 ppm
I20260927 22:41:18.160609 8909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:18 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548878, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:18.163698 8909 init.cc:377] Logged in from keytab as oryx/127.1.53.65@KRBTEST.COM (short username oryx)
I20260927 22:41:18.165303 8909 webserver.cc:460] Webserver started at http://127.1.53.65:43727/ using document root <none> and password file <none>
I20260927 22:41:18.166132 8909 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:18.166267 8909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:18.166600 8909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:18.169649 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "2543a1928c07483f9ae1ee7c0b24b527"
format_stamp: "Formatted at 2026-09-27 22:41:18 on dist-test-slave-pp80"
server_key: "a4e2bf80e386637be9e60a7207913383"
server_key_iv: "61d782ee282b1da0ea6b51737b3be91f"
server_key_version: "encryptionkey@0"
I20260927 22:41:18.170461 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "2543a1928c07483f9ae1ee7c0b24b527"
format_stamp: "Formatted at 2026-09-27 22:41:18 on dist-test-slave-pp80"
server_key: "a4e2bf80e386637be9e60a7207913383"
server_key_iv: "61d782ee282b1da0ea6b51737b3be91f"
server_key_version: "encryptionkey@0"
I20260927 22:41:18.175799 8909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:41:18.179379 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:18.180742 8909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:18.180918 8909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "2543a1928c07483f9ae1ee7c0b24b527"
format_stamp: "Formatted at 2026-09-27 22:41:18 on dist-test-slave-pp80"
server_key: "a4e2bf80e386637be9e60a7207913383"
server_key_iv: "61d782ee282b1da0ea6b51737b3be91f"
server_key_version: "encryptionkey@0"
I20260927 22:41:18.181142 8909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:18.207978 8909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:18.212066 8909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:18.212397 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:18.215152 8909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:18.215257 8909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:18.215420 8909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:18.215502 8909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:41:18 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548878, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:18.240465 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.215950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:46349 (local address 127.1.53.126:35005)
0927 22:41:18.216452 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:18.216458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:18.217736 (+ 1278us) server_negotiation.cc:407] Connection header received
0927 22:41:18.220057 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:18.220062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:18.220151 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:18.220298 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:18.223125 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.224087 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.227500 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.228055 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.228203 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.233740 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:18.233745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:18.233749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:18.233781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:18.236372 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.237374 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.237380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.237385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.237535 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.238126 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.238133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.238137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.238480 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:18.238642 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:18.239779 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0927 22:41:18.239994 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260927 22:41:18.242259 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.216636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.1.53.65:46349)
0927 22:41:18.217538 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:18.217599 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:18.219711 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:18.220658 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:18.220682 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:18.221654 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:18.222856 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.222899 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.224315 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.224325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.227311 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.227336 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.228715 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.228727 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.229062 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.230054 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:18.230091 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:18.233497 (+ 3406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:18.236517 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.236526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.236548 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.237168 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.237652 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.237657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.237662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.237966 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.238641 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:18.238657 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:18.239221 (+ 564us) client_negotiation.cc:769] Sending connection context
0927 22:41:18.239635 (+ 414us) client_negotiation.cc:240] Negotiation successful
0927 22:41:18.240329 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":415,"threads_started":1}
I20260927 22:41:18.267236 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.249486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.1.53.65:56037)
0927 22:41:18.250226 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:18.250251 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:18.250421 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:18.250813 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:18.250821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:18.251552 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:18.252010 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.252023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.253297 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.253305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.255501 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.255530 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.256834 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.256845 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.257128 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.257932 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:18.257960 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:18.259357 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:18.264320 (+ 4963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.264326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.264332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.264898 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.265420 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.265423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.265426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.265547 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.266326 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:18.266331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:18.266515 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 22:41:18.266644 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:41:18.266902 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":398,"threads_started":1}
I20260927 22:41:18.267618 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.249497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:56037 (local address 127.1.53.126:35005)
0927 22:41:18.249719 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:18.249728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:18.250355 (+ 627us) server_negotiation.cc:407] Connection header received
0927 22:41:18.250577 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:18.250581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:18.250656 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:18.250756 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:18.252171 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.253132 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.255679 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.256521 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.256668 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.259640 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:18.259650 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:18.259658 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:18.259706 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:18.264102 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.265060 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.265070 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.265075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.265263 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.265659 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.265667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.265673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.266161 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:18.266338 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:18.266805 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 22:41:18.267124 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:41:18.278421 8909 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:35421
I20260927 22:41:18.278450 9044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:35421 every 8 connection(s)
I20260927 22:41:18.279616 8909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:18.281810 9045 heartbeater.cc:347] Connected to a master server at 127.1.53.126:35005
I20260927 22:41:18.281883 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 8909
I20260927 22:41:18.282061 9045 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:18.282089 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:18.282693 9045 heartbeater.cc:511] Master 127.1.53.126:35005 requested a full tablet report, sending...
I20260927 22:41:18.282720 1237 external_mini_cluster.cc:1428] Setting key 8ec895aac9ac4951c3cc20582dbb19a9
I20260927 22:41:18.284850 8837 ts_manager.cc:194] Registered new tserver with Master: 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421)
I20260927 22:41:18.286343 8837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:46349
WARNING: no policy specified for oryx/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab.
Entry for principal oryx/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab.
I20260927 22:41:18.329609 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35005
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:18.661015 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:18.661334 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:18.661506 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:18.668617 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:18.668767 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:18.668924 9049 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:41:18.669073 9049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:18.679044 9049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:35005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:18.680392 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:18.681722 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:18.695183 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:18.695314 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:18.695840 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:18.696171 9049 server_base.cc:1034] running on GCE node
I20260927 22:41:18.696849 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:18.697724 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:18.699040 9049 hybrid_clock.cc:648] HybridClock initialized: now 1790548878698958 us; error 35 us; skew 500 ppm
I20260927 22:41:18.699440 9049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:18 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548878, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:18.703140 9049 init.cc:377] Logged in from keytab as oryx/127.1.53.66@KRBTEST.COM (short username oryx)
I20260927 22:41:18.705037 9049 webserver.cc:460] Webserver started at http://127.1.53.66:44607/ using document root <none> and password file <none>
I20260927 22:41:18.705855 9049 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:18.706012 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:18.706377 9049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:18.709648 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
format_stamp: "Formatted at 2026-09-27 22:41:18 on dist-test-slave-pp80"
server_key: "9e9e11a5e6319a0655a354e7c1ffa948"
server_key_iv: "8c6a07233af49d84c8f72ec122ccb3b8"
server_key_version: "encryptionkey@0"
I20260927 22:41:18.710533 9049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
format_stamp: "Formatted at 2026-09-27 22:41:18 on dist-test-slave-pp80"
server_key: "9e9e11a5e6319a0655a354e7c1ffa948"
server_key_iv: "8c6a07233af49d84c8f72ec122ccb3b8"
server_key_version: "encryptionkey@0"
I20260927 22:41:18.716313 9049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:41:18.720427 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:18.721933 9049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:41:18.722136 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
format_stamp: "Formatted at 2026-09-27 22:41:18 on dist-test-slave-pp80"
server_key: "9e9e11a5e6319a0655a354e7c1ffa948"
server_key_iv: "8c6a07233af49d84c8f72ec122ccb3b8"
server_key_version: "encryptionkey@0"
I20260927 22:41:18.722394 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:18.747164 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:18.751574 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:18.751804 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:18.754918 9049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:18.755051 9049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:18.755283 9049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:18.755502 9049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:41:18 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548878, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:18.783398 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.755384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:51135 (local address 127.1.53.126:35005)
0927 22:41:18.755560 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:18.755565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:18.757764 (+ 2199us) server_negotiation.cc:407] Connection header received
0927 22:41:18.760808 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:18.760815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:18.760925 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:18.761007 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:18.763898 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.765061 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.767962 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.768661 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.768825 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.774347 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:18.774358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:18.774367 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:18.774424 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:18.778384 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.779753 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.779761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.779767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.779936 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.780585 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.780595 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.780602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.781113 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:18.781288 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:18.782319 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0927 22:41:18.782653 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:41:18.784103 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.756437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.1.53.66:51135)
0927 22:41:18.757580 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:18.757632 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:18.760336 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:18.761672 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:18.761696 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:18.762595 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:18.763665 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.763700 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.765225 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.765231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.767770 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.767796 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.768972 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.768982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.769214 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.770176 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:18.770216 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:18.774049 (+ 3833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:18.778568 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.778586 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.778609 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.779574 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.780078 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.780082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.780085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.780423 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.781309 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:18.781322 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:18.781730 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 22:41:18.782156 (+ 426us) client_negotiation.cc:240] Negotiation successful
0927 22:41:18.782782 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":610,"threads_started":1}
I20260927 22:41:18.808178 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.791915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:41989 (local address 127.1.53.126:35005)
0927 22:41:18.792385 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:18.792396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:18.793224 (+ 828us) server_negotiation.cc:407] Connection header received
0927 22:41:18.793448 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:18.793455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:18.793535 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:18.793671 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:18.794698 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.795478 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.797485 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.798176 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.798357 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.801426 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:18.801436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:18.801445 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:18.801520 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:18.805243 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.805958 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.805971 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.805974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.806102 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:18.806512 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:18.806516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:18.806519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:18.806854 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:18.807025 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:18.807522 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 22:41:18.807756 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260927 22:41:18.808331 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:18.791946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.1.53.66:41989)
0927 22:41:18.793095 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:18.793114 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:18.793301 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:18.793705 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:18.793710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:18.794229 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:18.794540 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.794549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.795632 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.795638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.797309 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:18.797332 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:18.798432 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:18.798443 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:18.798786 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:18.799529 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:18.799558 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:18.801182 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:18.805384 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.805391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.805396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.805820 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.806223 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:18.806229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:18.806234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:18.806373 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:18.806988 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:18.807001 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:18.807224 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 22:41:18.807404 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 22:41:18.807765 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":851,"threads_started":1}
I20260927 22:41:18.813241 9049 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:35743
I20260927 22:41:18.813266 9183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:35743 every 8 connection(s)
I20260927 22:41:18.814563 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:18.816699 9184 heartbeater.cc:347] Connected to a master server at 127.1.53.126:35005
I20260927 22:41:18.816933 9184 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:18.817600 9184 heartbeater.cc:511] Master 127.1.53.126:35005 requested a full tablet report, sending...
I20260927 22:41:18.818995 8837 ts_manager.cc:194] Registered new tserver with Master: 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743)
I20260927 22:41:18.819803 8837 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:51135
I20260927 22:41:18.822417 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 9049
I20260927 22:41:18.822635 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:18.823117 1237 external_mini_cluster.cc:1428] Setting key b4b43b8fcc1bb02c7f897ecdebd58362
WARNING: no policy specified for oryx/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab.
Entry for principal oryx/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab.
I20260927 22:41:18.872790 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:35005
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:19.206936 9188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:19.207227 9188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:19.207325 9188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:19.215029 9188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:19.215174 9188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:19.215265 9188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:41:19.215358 9188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:19.225455 9188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.53.126:35005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:19.226946 9188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:19.228300 9188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:19.241562 9196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:19.241585 9194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:19.241536 9193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:19.242722 9188 server_base.cc:1034] running on GCE node
I20260927 22:41:19.243323 9188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:19.244217 9188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:19.245513 9188 hybrid_clock.cc:648] HybridClock initialized: now 1790548879245422 us; error 67 us; skew 500 ppm
I20260927 22:41:19.245891 9188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:19 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548879, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:19.249208 9188 init.cc:377] Logged in from keytab as oryx/127.1.53.67@KRBTEST.COM (short username oryx)
I20260927 22:41:19.250690 9188 webserver.cc:460] Webserver started at http://127.1.53.67:33759/ using document root <none> and password file <none>
I20260927 22:41:19.251375 9188 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:19.251505 9188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:19.251823 9188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:19.255005 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "56cb69e7e84a463d8d590281fbb198f7"
format_stamp: "Formatted at 2026-09-27 22:41:19 on dist-test-slave-pp80"
server_key: "9f84f0b30244c3225de22efe44f68623"
server_key_iv: "89c28f37efb1acc67edba60970c6cb9c"
server_key_version: "encryptionkey@0"
I20260927 22:41:19.255786 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "56cb69e7e84a463d8d590281fbb198f7"
format_stamp: "Formatted at 2026-09-27 22:41:19 on dist-test-slave-pp80"
server_key: "9f84f0b30244c3225de22efe44f68623"
server_key_iv: "89c28f37efb1acc67edba60970c6cb9c"
server_key_version: "encryptionkey@0"
I20260927 22:41:19.260818 9188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:41:19.264575 9203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:19.266047 9188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:41:19.266245 9188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "56cb69e7e84a463d8d590281fbb198f7"
format_stamp: "Formatted at 2026-09-27 22:41:19 on dist-test-slave-pp80"
server_key: "9f84f0b30244c3225de22efe44f68623"
server_key_iv: "89c28f37efb1acc67edba60970c6cb9c"
server_key_version: "encryptionkey@0"
I20260927 22:41:19.266511 9188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:19.286255 9188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:19.289132 9045 heartbeater.cc:503] Master 127.1.53.126:35005 was elected leader, sending a full tablet report...
I20260927 22:41:19.291426 9188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:19.291661 9188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:19.294502 9188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:19.294632 9188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:19.294772 9188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:19.294891 9188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 27 22:41:19 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548879, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:19.315613 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.294764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:59749 (local address 127.1.53.126:35005)
0927 22:41:19.294968 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.294974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.296502 (+ 1528us) server_negotiation.cc:407] Connection header received
0927 22:41:19.298496 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.298502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.298586 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.298702 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:19.300642 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.301781 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.303364 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.303774 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.303903 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.308980 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:19.308985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:19.308988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:19.309021 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:19.311635 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:19.312428 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:19.312437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:19.312445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:19.312575 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:19.313017 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:19.313022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:19.313029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:19.313286 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:19.313474 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.314625 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.315017 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:41:19.316102 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.295313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.1.53.67:59749)
0927 22:41:19.296320 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.296372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.298233 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.298933 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.298941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.299451 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:19.300368 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.300399 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.301953 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.301957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.303224 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.303235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.304010 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.304017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.304367 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.305131 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:19.305171 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:19.308749 (+ 3578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:19.311789 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:19.311799 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:19.311817 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:19.312296 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:19.312685 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:19.312691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:19.312695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:19.312894 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:19.313757 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:19.313762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:19.314098 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.314391 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.314799 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":404,"threads_started":1}
I20260927 22:41:19.344900 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.325663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.1.53.67:50093)
0927 22:41:19.329076 (+ 3413us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.329101 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.329292 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.329786 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.329789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.330234 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:19.330564 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.330574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.332541 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.332548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.334204 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.334216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.335087 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.335093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.335277 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.335944 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:19.335961 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:19.336930 (+ 969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:19.341965 (+ 5035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:19.341969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:19.341972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:19.342351 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:19.342879 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:19.342882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:19.342884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:19.343002 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:19.343810 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:19.343816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:19.343982 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.344149 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.344504 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3210,"thread_start_us":3049,"threads_started":1}
I20260927 22:41:19.345356 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.326097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:50093 (local address 127.1.53.126:35005)
0927 22:41:19.326288 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.326295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.329230 (+ 2935us) server_negotiation.cc:407] Connection header received
0927 22:41:19.329462 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.329472 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.329611 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.329775 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:19.330730 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.332323 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.334359 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.334887 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.335048 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.337216 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:19.337221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:19.337225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:19.337265 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:19.341819 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:19.342506 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:19.342516 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:19.342522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:19.342732 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:19.343145 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:19.343153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:19.343159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:19.343633 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:19.343831 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.344430 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.344773 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:41:19.362660 9188 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39071
I20260927 22:41:19.362699 9322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39071 every 8 connection(s)
I20260927 22:41:19.363998 9188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:19.365506 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 9188
I20260927 22:41:19.365694 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:19.366300 1237 external_mini_cluster.cc:1428] Setting key b5aeda99286ee90877c804d46edcac09
I20260927 22:41:19.366488 9323 heartbeater.cc:347] Connected to a master server at 127.1.53.126:35005
I20260927 22:41:19.366677 9323 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:19.367561 9323 heartbeater.cc:511] Master 127.1.53.126:35005 requested a full tablet report, sending...
I20260927 22:41:19.369227 8838 ts_manager.cc:194] Registered new tserver with Master: 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:19.370250 8838 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:59749
I20260927 22:41:19.381951 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:41:19 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:41:19 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:19 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:41:19.433341 8933 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:19.422558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.1.53.126:35005)
0927 22:41:19.422809 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.422816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.423024 (+ 208us) server_negotiation.cc:407] Connection header received
0927 22:41:19.423239 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.423252 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.423314 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.423419 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:19.424833 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.425817 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.427140 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.427578 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.427717 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.433116 (+ 5399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58024: BlockingRecv error: recv got EOF from 127.0.0.1:58024 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 22:41:19 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:19.450685 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.437233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.1.53.126:35005)
0927 22:41:19.437462 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.437468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.437701 (+ 233us) server_negotiation.cc:407] Connection header received
0927 22:41:19.437903 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.437907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.437986 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.438090 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:19.439324 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.440099 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.441400 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.441799 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.441947 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.445723 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:19.445729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:19.445732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:19.445762 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:19.448113 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:19.448838 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:19.448844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:19.448848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:19.448969 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:19.449339 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:19.449344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:19.449348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:19.449609 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:19.449882 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.450155 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.450365 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:41:19.454662 8838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:19.461591 8838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:19.485219 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.476857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.1.53.66:35743)
0927 22:41:19.477664 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.477672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.477697 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:41:19.477783 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.477789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.478022 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.478209 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:19.478873 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.480693 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.482528 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.483921 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.484134 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.484394 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.484640 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.484804 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":754,"threads_started":1}
I20260927 22:41:19.485177 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.474807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:35421 (local address 127.0.0.1:44190)
0927 22:41:19.476486 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.476507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.476621 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.477364 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.477375 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.477420 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:19.477818 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.477830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.479894 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.479899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.482562 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.482575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.484354 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.484360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.484517 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.484570 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.484639 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.484765 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"thread_start_us":1303,"threads_started":1}
I20260927 22:41:19.485280 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.476610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:35743 (local address 127.0.0.1:44076)
0927 22:41:19.477089 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.477108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.477224 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.478220 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.478226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.478251 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:19.478695 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.478703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.480839 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.480846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.482370 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.482382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.484164 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.484171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.484378 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.484428 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.484578 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.484764 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":37,"thread_start_us":509,"threads_started":1}
I20260927 22:41:19.485864 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.474634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39071 (local address 127.0.0.1:54416)
0927 22:41:19.475481 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.475541 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.475765 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.478400 (+ 2635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.478404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.478431 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:19.478889 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.478901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.480912 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.480918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.483044 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.483057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.485004 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.485012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.485192 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.485236 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.485337 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.485462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":5,"thread_start_us":368,"threads_started":1}
I20260927 22:41:19.486249 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.475352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.1.53.65:35421)
0927 22:41:19.476710 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.476721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.476754 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:41:19.476850 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.476857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.477118 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.478008 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:19.478189 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.479757 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.482717 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.484098 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.485495 (+ 1397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.485692 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.485756 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.485963 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":1364,"threads_started":1}
I20260927 22:41:19.487008 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.474932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.1.53.67:39071)
0927 22:41:19.477773 (+ 2841us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.477784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.477817 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:41:19.477916 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.477925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.478182 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.479087 (+ 905us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:19.479394 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.480785 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.483222 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.484755 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.486212 (+ 1457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.486445 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.486513 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.486617 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2668,"thread_start_us":2476,"threads_started":1}
I20260927 22:41:19.489310 9114 tablet_service.cc:1467] Processing CreateTablet for tablet a4c026bd846d49558d23b87521243a19 (DEFAULT_TABLE table=test-table [id=10de13c3ae8146abb886e0faf5c106fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:19.489777 8975 tablet_service.cc:1467] Processing CreateTablet for tablet a4c026bd846d49558d23b87521243a19 (DEFAULT_TABLE table=test-table [id=10de13c3ae8146abb886e0faf5c106fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:19.490129 9257 tablet_service.cc:1467] Processing CreateTablet for tablet a4c026bd846d49558d23b87521243a19 (DEFAULT_TABLE table=test-table [id=10de13c3ae8146abb886e0faf5c106fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:19.491196 8975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c026bd846d49558d23b87521243a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:19.491284 9114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c026bd846d49558d23b87521243a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:19.491377 9257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c026bd846d49558d23b87521243a19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:19.502774 9347 tablet_bootstrap.cc:492] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Bootstrap starting.
I20260927 22:41:19.505908 9348 tablet_bootstrap.cc:492] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Bootstrap starting.
I20260927 22:41:19.506670 9347 tablet_bootstrap.cc:654] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:19.507355 9349 tablet_bootstrap.cc:492] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Bootstrap starting.
I20260927 22:41:19.508476 9347 log.cc:824] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:19.509829 9348 tablet_bootstrap.cc:654] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:19.510769 9349 tablet_bootstrap.cc:654] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:19.511377 9348 log.cc:824] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:19.512053 9349 log.cc:824] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:19.514572 9349 tablet_bootstrap.cc:492] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: No bootstrap required, opened a new log
I20260927 22:41:19.514722 9348 tablet_bootstrap.cc:492] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: No bootstrap required, opened a new log
I20260927 22:41:19.514741 9347 tablet_bootstrap.cc:492] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: No bootstrap required, opened a new log
I20260927 22:41:19.514892 9349 ts_tablet_manager.cc:1397] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:41:19.514998 9348 ts_tablet_manager.cc:1397] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:41:19.514998 9347 ts_tablet_manager.cc:1397] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 22:41:19.522446 9349 raft_consensus.cc:348] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.522815 9349 raft_consensus.cc:374] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:19.522907 9349 raft_consensus.cc:729] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Initialized, Role: FOLLOWER
I20260927 22:41:19.523561 9349 consensus_queue.cc:260] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.525310 9349 ts_tablet_manager.cc:1428] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 22:41:19.525728 9348 raft_consensus.cc:348] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.525722 9347 raft_consensus.cc:348] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.526144 9348 raft_consensus.cc:374] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:19.526144 9347 raft_consensus.cc:374] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:19.526250 9348 raft_consensus.cc:729] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Initialized, Role: FOLLOWER
I20260927 22:41:19.526248 9347 raft_consensus.cc:729] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b0835765d8474e9f66bb7ae4e0b80d, State: Initialized, Role: FOLLOWER
I20260927 22:41:19.526844 9348 consensus_queue.cc:260] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.526872 9347 consensus_queue.cc:260] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.529383 9323 heartbeater.cc:503] Master 127.1.53.126:35005 was elected leader, sending a full tablet report...
I20260927 22:41:19.530439 9348 ts_tablet_manager.cc:1428] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260927 22:41:19.531549 9184 heartbeater.cc:503] Master 127.1.53.126:35005 was elected leader, sending a full tablet report...
W20260927 22:41:19.531931 9046 tablet.cc:2369] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:19.532202 9347 ts_tablet_manager.cc:1428] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
W20260927 22:41:19.566766 9185 tablet.cc:2369] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:41:19.615831 9324 tablet.cc:2369] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:19.677731 9354 raft_consensus.cc:482] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:19.678026 9354 raft_consensus.cc:504] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.679806 9354 leader_election.cc:288] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421), 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743)
I20260927 22:41:19.686684 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.680128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:35421 (local address 127.1.53.67:35063)
0927 22:41:19.680362 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.680382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.680518 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.681070 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.681082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.681108 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:19.681305 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.681318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.682573 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.682580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.684590 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.684602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.686019 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.686024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.686237 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.686262 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.686337 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.686486 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":35}
I20260927 22:41:19.686803 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.680173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:35063 (local address 127.1.53.65:35421)
0927 22:41:19.680385 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.680390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.680439 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:41:19.680680 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.680686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.680778 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.680880 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:19.681441 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.682422 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.684749 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.685775 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.685998 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.686106 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.686394 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.686538 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:41:19.687915 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.680788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:58641 (local address 127.1.53.66:35743)
0927 22:41:19.680974 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.680979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.681290 (+ 311us) server_negotiation.cc:407] Connection header received
0927 22:41:19.681396 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.681404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.681491 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.681561 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:19.682181 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.683420 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.685461 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.686799 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.686948 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.687057 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.687629 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.687723 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:41:19.687886 8999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c026bd846d49558d23b87521243a19" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2543a1928c07483f9ae1ee7c0b24b527" is_pre_election: true
I20260927 22:41:19.688133 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.680555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:35743 (local address 127.1.53.67:58641)
0927 22:41:19.681182 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:19.681196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:19.681295 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:19.681609 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:19.681613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:19.681636 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:19.681986 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.681996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.683597 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.683602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.685294 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:19.685305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.687204 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.687209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:19.687434 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.687477 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:41:19.687588 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:41:19.687718 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":49,"thread_start_us":338,"threads_started":1}
I20260927 22:41:19.688577 8999 raft_consensus.cc:2454] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 0.
I20260927 22:41:19.689270 9207 leader_election.cc:302] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543a1928c07483f9ae1ee7c0b24b527, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:19.689189 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c026bd846d49558d23b87521243a19" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" is_pre_election: true
I20260927 22:41:19.689707 9354 raft_consensus.cc:2781] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:19.689703 9134 raft_consensus.cc:2454] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 0.
I20260927 22:41:19.689893 9354 raft_consensus.cc:482] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:19.690021 9354 raft_consensus.cc:3037] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:19.692487 9354 raft_consensus.cc:504] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.693037 9354 leader_election.cc:288] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 election: Requested vote from peers 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421), 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743)
I20260927 22:41:19.693491 8999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c026bd846d49558d23b87521243a19" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2543a1928c07483f9ae1ee7c0b24b527"
I20260927 22:41:19.693536 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c026bd846d49558d23b87521243a19" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
I20260927 22:41:19.693683 8999 raft_consensus.cc:3037] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:19.693737 9134 raft_consensus.cc:3037] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:19.695871 9134 raft_consensus.cc:2454] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 1.
I20260927 22:41:19.695968 8999 raft_consensus.cc:2454] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 1.
I20260927 22:41:19.696297 9207 leader_election.cc:302] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b0835765d8474e9f66bb7ae4e0b80d, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:19.696550 9354 raft_consensus.cc:2781] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:19.697026 9354 raft_consensus.cc:686] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 LEADER]: Becoming Leader. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Running, Role: LEADER
I20260927 22:41:19.697505 9354 consensus_queue.cc:237] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } }
I20260927 22:41:19.702379 8837 catalog_manager.cc:5495] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67). New cstate: current_term: 1 leader_uuid: "56cb69e7e84a463d8d590281fbb198f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } health_report { overall_health: UNKNOWN } } }
I20260927 22:41:19.727932 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:19.722653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.1.53.67:39071)
0927 22:41:19.722881 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:19.722887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:19.722906 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:41:19.723153 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:19.723158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:19.723241 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:19.723395 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:19.723748 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.724505 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.725830 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:19.726254 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:19.726387 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:19.726863 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:19.727381 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:19.727547 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:19.727596 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 22:41:19.727675 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:41:19.735775 8999 raft_consensus.cc:1261] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Refusing update from remote peer 56cb69e7e84a463d8d590281fbb198f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:19.735776 9134 raft_consensus.cc:1261] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Refusing update from remote peer 56cb69e7e84a463d8d590281fbb198f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:19.736662 9357 consensus_queue.cc:1035] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:19.737038 9354 consensus_queue.cc:1035] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:19.755385 9362 mvcc.cc:205] Tried to move back new op lower bound from 7334088211388391424 to 7334088211244523520. Current Snapshot: MvccSnapshot[applied={T|T < 7334088211388391424}]
I20260927 22:41:19.769048 8838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:19.769335 8838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:19.773234 8838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Sending DeleteTablet for 3 replicas of tablet a4c026bd846d49558d23b87521243a19
I20260927 22:41:19.774539 9257 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c026bd846d49558d23b87521243a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:19 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54416
I20260927 22:41:19.774705 9114 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c026bd846d49558d23b87521243a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:19 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44076
I20260927 22:41:19.775439 8975 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c026bd846d49558d23b87521243a19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:19 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44190
I20260927 22:41:19.775794 8838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:19.776688 8838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:19.787734 9256 tablet_service.cc:1467] Processing CreateTablet for tablet ce8f4624964647f1a875820534506075 (DEFAULT_TABLE table=test-table [id=3e49996d06204a0bafcc974ac4863cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:19.788259 9113 tablet_service.cc:1467] Processing CreateTablet for tablet ce8f4624964647f1a875820534506075 (DEFAULT_TABLE table=test-table [id=3e49996d06204a0bafcc974ac4863cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:19.788597 9256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8f4624964647f1a875820534506075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:19.789019 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8f4624964647f1a875820534506075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:19.793367 8974 tablet_service.cc:1467] Processing CreateTablet for tablet ce8f4624964647f1a875820534506075 (DEFAULT_TABLE table=test-table [id=3e49996d06204a0bafcc974ac4863cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:19.794119 8974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8f4624964647f1a875820534506075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:19.798071 9348 tablet_bootstrap.cc:492] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Bootstrap starting.
I20260927 22:41:19.800632 9372 tablet_replica.cc:330] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: stopping tablet replica
I20260927 22:41:19.801187 9372 raft_consensus.cc:2229] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:19.801748 9372 raft_consensus.cc:2258] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:19.801858 9348 tablet_bootstrap.cc:654] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:19.802644 9371 tablet_replica.cc:330] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: stopping tablet replica
I20260927 22:41:19.803076 9371 raft_consensus.cc:2229] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:19.803761 9371 raft_consensus.cc:2258] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:19.806173 9372 ts_tablet_manager.cc:1905] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:19.810259 9371 ts_tablet_manager.cc:1905] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:19.822468 9349 tablet_bootstrap.cc:492] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Bootstrap starting.
I20260927 22:41:19.824796 9349 tablet_bootstrap.cc:654] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:19.824859 9370 tablet_replica.cc:330] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: stopping tablet replica
I20260927 22:41:19.825380 9370 raft_consensus.cc:2229] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:19.825525 9347 tablet_bootstrap.cc:492] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Bootstrap starting.
I20260927 22:41:19.827787 9347 tablet_bootstrap.cc:654] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:19.831188 9349 tablet_bootstrap.cc:492] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: No bootstrap required, opened a new log
I20260927 22:41:19.831364 9349 ts_tablet_manager.cc:1397] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:41:19.832482 9349 raft_consensus.cc:348] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.832792 9349 raft_consensus.cc:374] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:19.834055 9370 raft_consensus.cc:2258] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:19.834363 9349 raft_consensus.cc:729] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Initialized, Role: FOLLOWER
I20260927 22:41:19.834573 9348 tablet_bootstrap.cc:492] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: No bootstrap required, opened a new log
I20260927 22:41:19.834726 9348 ts_tablet_manager.cc:1397] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.010s
I20260927 22:41:19.834816 9349 consensus_queue.cc:260] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.835673 9349 ts_tablet_manager.cc:1428] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260927 22:41:19.835633 9348 raft_consensus.cc:348] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.835865 9348 raft_consensus.cc:374] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:19.835991 9348 raft_consensus.cc:729] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Initialized, Role: FOLLOWER
I20260927 22:41:19.836306 9348 consensus_queue.cc:260] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.837616 9348 ts_tablet_manager.cc:1428] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:19.842053 9370 ts_tablet_manager.cc:1905] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:19.854254 9372 ts_tablet_manager.cc:1918] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:19.854554 9372 log.cc:1196] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/a4c026bd846d49558d23b87521243a19
I20260927 22:41:19.855271 9372 ts_tablet_manager.cc:1939] T a4c026bd846d49558d23b87521243a19 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting consensus metadata
I20260927 22:41:19.857352 8825 catalog_manager.cc:4841] TS 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421): tablet a4c026bd846d49558d23b87521243a19 (table test-table [id=10de13c3ae8146abb886e0faf5c106fb]) successfully deleted
I20260927 22:41:19.859300 9371 ts_tablet_manager.cc:1918] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:19.866006 9371 log.cc:1196] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/a4c026bd846d49558d23b87521243a19
I20260927 22:41:19.866714 9371 ts_tablet_manager.cc:1939] T a4c026bd846d49558d23b87521243a19 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting consensus metadata
I20260927 22:41:19.867440 9347 tablet_bootstrap.cc:492] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: No bootstrap required, opened a new log
I20260927 22:41:19.867661 9347 ts_tablet_manager.cc:1397] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.031s
I20260927 22:41:19.868603 9347 raft_consensus.cc:348] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.868927 9347 raft_consensus.cc:374] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:19.869016 8825 catalog_manager.cc:4841] TS 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743): tablet a4c026bd846d49558d23b87521243a19 (table test-table [id=10de13c3ae8146abb886e0faf5c106fb]) successfully deleted
I20260927 22:41:19.869082 9347 raft_consensus.cc:729] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b0835765d8474e9f66bb7ae4e0b80d, State: Initialized, Role: FOLLOWER
I20260927 22:41:19.869447 9347 consensus_queue.cc:260] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.870081 9347 ts_tablet_manager.cc:1428] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:41:19.870724 9370 ts_tablet_manager.cc:1918] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:19.870901 9370 log.cc:1196] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/a4c026bd846d49558d23b87521243a19
I20260927 22:41:19.871527 9370 ts_tablet_manager.cc:1939] T a4c026bd846d49558d23b87521243a19 P 56cb69e7e84a463d8d590281fbb198f7: Deleting consensus metadata
I20260927 22:41:19.873358 8822 catalog_manager.cc:4841] TS 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071): tablet a4c026bd846d49558d23b87521243a19 (table test-table [id=10de13c3ae8146abb886e0faf5c106fb]) successfully deleted
I20260927 22:41:19.939708 9365 raft_consensus.cc:482] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:19.940021 9365 raft_consensus.cc:504] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.940853 9365 leader_election.cc:288] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421), 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743)
I20260927 22:41:19.941478 8999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8f4624964647f1a875820534506075" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2543a1928c07483f9ae1ee7c0b24b527" is_pre_election: true
I20260927 22:41:19.941527 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8f4624964647f1a875820534506075" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" is_pre_election: true
I20260927 22:41:19.941730 8999 raft_consensus.cc:2454] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 0.
I20260927 22:41:19.941794 9134 raft_consensus.cc:2454] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 0.
I20260927 22:41:19.942205 9207 leader_election.cc:302] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543a1928c07483f9ae1ee7c0b24b527, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:19.942541 9365 raft_consensus.cc:2781] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:19.942667 9365 raft_consensus.cc:482] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:19.942757 9365 raft_consensus.cc:3037] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:19.945966 9365 raft_consensus.cc:504] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.946631 9365 leader_election.cc:288] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 election: Requested vote from peers 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421), 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743)
I20260927 22:41:19.947136 8999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8f4624964647f1a875820534506075" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2543a1928c07483f9ae1ee7c0b24b527"
I20260927 22:41:19.947199 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8f4624964647f1a875820534506075" candidate_uuid: "56cb69e7e84a463d8d590281fbb198f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
I20260927 22:41:19.947346 8999 raft_consensus.cc:3037] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:19.947427 9134 raft_consensus.cc:3037] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:19.950066 9134 raft_consensus.cc:2454] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 1.
I20260927 22:41:19.950275 8999 raft_consensus.cc:2454] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cb69e7e84a463d8d590281fbb198f7 in term 1.
I20260927 22:41:19.950501 9207 leader_election.cc:302] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b0835765d8474e9f66bb7ae4e0b80d, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:19.950815 9365 raft_consensus.cc:2781] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:19.951208 9365 raft_consensus.cc:686] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 LEADER]: Becoming Leader. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Running, Role: LEADER
I20260927 22:41:19.951516 9365 consensus_queue.cc:237] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:19.954280 8838 catalog_manager.cc:5495] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67). New cstate: current_term: 1 leader_uuid: "56cb69e7e84a463d8d590281fbb198f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } health_report { overall_health: HEALTHY } } }
Sep 27 22:41:20 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548877, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.126@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:20.021699 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.006718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.1.53.126:35005)
0927 22:41:20.008655 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.008659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.008674 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:41:20.008722 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.008726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.008777 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.008901 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:20.009976 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.010752 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.011952 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.012432 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.012574 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.016183 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:20.016189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:20.016192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:20.016219 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:20.018912 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:20.019612 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:20.019619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:20.019622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:20.019741 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:20.020112 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:20.020120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:20.020124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:20.020464 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:20.020613 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.021001 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.021354 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":1845,"thread_start_us":1706,"threads_started":1}
I20260927 22:41:20.021706 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.006574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:35005 (local address 127.0.0.1:58046)
0927 22:41:20.007150 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:20.007167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:20.007277 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:20.008932 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:20.008936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:20.009601 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:20.009776 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.009786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.010905 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.010912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.011806 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.011817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.012593 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.012599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.012798 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.013512 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:20.013529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:20.015983 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:20.019073 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:20.019086 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:20.019094 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:20.019494 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:20.019854 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:20.019858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:20.019862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:20.020000 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:20.020603 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:20.020612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:20.020756 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 22:41:20.020929 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 22:41:20.021248 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":4,"thread_start_us":270,"threads_started":1}
I20260927 22:41:20.027294 8838 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:58046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:41:20.028368 8838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:20.035808 8974 tablet_service.cc:1467] Processing CreateTablet for tablet 87a9948502c9404ba4bebf4a88cfb9bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb10d1845f2e4947a4a6a5450497038d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:41:20.035825 9113 tablet_service.cc:1467] Processing CreateTablet for tablet 87a9948502c9404ba4bebf4a88cfb9bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb10d1845f2e4947a4a6a5450497038d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:41:20.035961 9257 tablet_service.cc:1467] Processing CreateTablet for tablet 87a9948502c9404ba4bebf4a88cfb9bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb10d1845f2e4947a4a6a5450497038d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:41:20.036403 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a9948502c9404ba4bebf4a88cfb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.036496 9257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a9948502c9404ba4bebf4a88cfb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.036640 8974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a9948502c9404ba4bebf4a88cfb9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.043905 9348 tablet_bootstrap.cc:492] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: Bootstrap starting.
I20260927 22:41:20.043892 9347 tablet_bootstrap.cc:492] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: Bootstrap starting.
I20260927 22:41:20.045718 9347 tablet_bootstrap.cc:654] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.045724 9348 tablet_bootstrap.cc:654] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.046445 9349 tablet_bootstrap.cc:492] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: Bootstrap starting.
I20260927 22:41:20.048216 9349 tablet_bootstrap.cc:654] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.048704 9347 tablet_bootstrap.cc:492] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: No bootstrap required, opened a new log
I20260927 22:41:20.048861 9347 ts_tablet_manager.cc:1397] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:41:20.048962 9348 tablet_bootstrap.cc:492] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: No bootstrap required, opened a new log
I20260927 22:41:20.049165 9348 ts_tablet_manager.cc:1397] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:41:20.049592 9347 raft_consensus.cc:348] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.049840 9347 raft_consensus.cc:374] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.049923 9347 raft_consensus.cc:729] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b0835765d8474e9f66bb7ae4e0b80d, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.050200 9347 consensus_queue.cc:260] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.050228 9348 raft_consensus.cc:348] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.050576 9348 raft_consensus.cc:374] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.050669 9348 raft_consensus.cc:729] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.050637 9355 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.050998 9355 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.050954 9348 consensus_queue.cc:260] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.051716 9348 ts_tablet_manager.cc:1428] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:41:20.053778 9347 ts_tablet_manager.cc:1428] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:41:20.054343 9365 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.054816 9365 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.061868 9349 tablet_bootstrap.cc:492] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: No bootstrap required, opened a new log
I20260927 22:41:20.062155 9349 ts_tablet_manager.cc:1397] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260927 22:41:20.062885 9349 raft_consensus.cc:348] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.063162 9349 raft_consensus.cc:374] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.063258 9349 raft_consensus.cc:729] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.063505 9349 consensus_queue.cc:260] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.064172 9349 ts_tablet_manager.cc:1428] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:41:20.063974 9353 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.064410 9353 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.243283 9353 raft_consensus.cc:482] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:20.243494 9353 raft_consensus.cc:504] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.247442 9353 leader_election.cc:288] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743), 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:20.255143 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.246045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:58159 (local address 127.1.53.67:39071)
0927 22:41:20.246567 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.246574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.246596 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:41:20.246677 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.246684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.246770 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.246874 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:20.247829 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.249169 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.252554 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.253415 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.253557 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.253659 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.254565 (+ 906us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.254731 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":904,"threads_started":1}
I20260927 22:41:20.255497 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.246471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:35743 (local address 127.1.53.65:53897)
0927 22:41:20.247028 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:20.247056 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:20.247172 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:20.247880 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:20.247885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:20.247915 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:20.248352 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.248364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.249846 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.249851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.252331 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.252349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.254096 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.254102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.254277 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.254319 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:41:20.254402 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:41:20.254523 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":96,"thread_start_us":800,"threads_started":1}
I20260927 22:41:20.255685 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.246799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:53897 (local address 127.1.53.66:35743)
0927 22:41:20.247314 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.247321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.247338 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:41:20.247408 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.247412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.247488 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.247588 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:20.248557 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.249658 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.252540 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.253733 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.253860 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.253948 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.255155 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.255323 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":825,"threads_started":1}
I20260927 22:41:20.255534 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.245637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39071 (local address 127.1.53.65:58159)
0927 22:41:20.246269 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:20.246295 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:20.246469 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:20.247291 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:20.247296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:20.247319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:20.247685 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.247697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.249761 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.249769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.252330 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.252349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.253593 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.253599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.253776 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.253829 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:41:20.253936 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:41:20.254114 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":200,"thread_start_us":1494,"threads_started":1}
I20260927 22:41:20.256740 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a9948502c9404ba4bebf4a88cfb9bd" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" is_pre_election: true
I20260927 22:41:20.256891 9278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a9948502c9404ba4bebf4a88cfb9bd" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cb69e7e84a463d8d590281fbb198f7" is_pre_election: true
I20260927 22:41:20.257043 9134 raft_consensus.cc:2454] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 0.
I20260927 22:41:20.257267 9278 raft_consensus.cc:2454] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 0.
I20260927 22:41:20.258277 8928 leader_election.cc:302] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b0835765d8474e9f66bb7ae4e0b80d, 2543a1928c07483f9ae1ee7c0b24b527; no voters:
I20260927 22:41:20.258733 9353 raft_consensus.cc:2781] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:20.258857 9353 raft_consensus.cc:482] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:20.258957 9353 raft_consensus.cc:3037] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:20.261714 9353 raft_consensus.cc:504] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.262449 9353 leader_election.cc:288] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 election: Requested vote from peers 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743), 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:20.262812 9134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a9948502c9404ba4bebf4a88cfb9bd" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
I20260927 22:41:20.262986 9278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a9948502c9404ba4bebf4a88cfb9bd" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cb69e7e84a463d8d590281fbb198f7"
I20260927 22:41:20.263067 9134 raft_consensus.cc:3037] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:20.263203 9278 raft_consensus.cc:3037] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:20.266002 9278 raft_consensus.cc:2454] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 1.
I20260927 22:41:20.266319 9134 raft_consensus.cc:2454] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 1.
I20260927 22:41:20.266520 8925 leader_election.cc:302] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543a1928c07483f9ae1ee7c0b24b527, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:20.266865 9353 raft_consensus.cc:2781] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:20.267299 9353 raft_consensus.cc:686] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 LEADER]: Becoming Leader. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Running, Role: LEADER
I20260927 22:41:20.267710 9353 consensus_queue.cc:237] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.269680 9391 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: TxnStatusTablet state changed. Reason: New leader 2543a1928c07483f9ae1ee7c0b24b527. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.270098 9391 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:41:20.270764 9393 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:41:20.270990 9393 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:41:20.271647 8838 catalog_manager.cc:5495] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } health_report { overall_health: UNKNOWN } } }
I20260927 22:41:20.303709 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.292284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.1.53.65:35421)
0927 22:41:20.296738 (+ 4454us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.296744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.296768 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:41:20.296863 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.296868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.296981 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.297159 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:20.297544 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.298831 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.300693 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.301432 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.301570 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.302110 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.302894 (+ 784us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.303058 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.303199 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.303286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4312,"thread_start_us":4164,"threads_started":1}
I20260927 22:41:20.303967 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.292203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:35421 (local address 127.0.0.1:44194)
0927 22:41:20.292635 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:20.292661 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:20.292831 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:20.297203 (+ 4372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:20.297208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:20.297221 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:41:20.297400 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.297410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.298972 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.298978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.300551 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.300565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.301608 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.301614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.301796 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.301866 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.303040 (+ 1174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.303062 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:41:20.303141 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:41:20.303298 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260927 22:41:20.309082 9134 raft_consensus.cc:1261] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Refusing update from remote peer 2543a1928c07483f9ae1ee7c0b24b527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:20.309180 9278 raft_consensus.cc:1261] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Refusing update from remote peer 2543a1928c07483f9ae1ee7c0b24b527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:20.309854 9391 consensus_queue.cc:1035] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:20.310170 9353 consensus_queue.cc:1035] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:20.319077 9365 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: TxnStatusTablet state changed. Reason: New leader 2543a1928c07483f9ae1ee7c0b24b527. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.319393 9365 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.321908 9355 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: TxnStatusTablet state changed. Reason: New leader 2543a1928c07483f9ae1ee7c0b24b527. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.322209 9355 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.332882 9355 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.333231 9355 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 18b0835765d8474e9f66bb7ae4e0b80d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.342876 9365 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.343170 9365 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 56cb69e7e84a463d8d590281fbb198f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:41:20.343631 9353 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.343997 9353 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:41:20.346716 9407 tablet_replica.cc:439] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } } }
I20260927 22:41:20.347065 9407 tablet_replica.cc:442] T 87a9948502c9404ba4bebf4a88cfb9bd P 2543a1928c07483f9ae1ee7c0b24b527: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:41:20.350268 9365 consensus_queue.cc:1035] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:20.357739 9357 consensus_queue.cc:1035] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:20.369443 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.354274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:38917 (local address 127.1.53.65:35421)
0927 22:41:20.354490 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.354500 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.363973 (+ 9473us) server_negotiation.cc:407] Connection header received
0927 22:41:20.364195 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.364201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.364326 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.364417 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:20.365092 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.365833 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.367234 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.367838 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.367952 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.368647 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.368777 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.368839 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.369069 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.369145 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:41:20.370270 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.353861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:35421 (local address 127.1.53.67:38917)
0927 22:41:20.363862 (+ 10001us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:20.363879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:20.364061 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:20.364631 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:20.364639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:20.364652 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:41:20.364940 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.364948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.365976 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.365984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.367097 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.367109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.368123 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.368130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.368359 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.368426 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.368917 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.368926 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:41:20.369696 (+ 770us) client_negotiation.cc:240] Negotiation successful
0927 22:41:20.369845 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9843,"thread_start_us":10307,"threads_started":1}
I20260927 22:41:20.411600 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.401714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:33099 (local address 127.1.53.67:39071)
0927 22:41:20.401877 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.401883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.404951 (+ 3068us) server_negotiation.cc:407] Connection header received
0927 22:41:20.405202 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.405207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.405298 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.405363 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:20.406724 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.407598 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.409196 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.409819 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.409923 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.410874 (+ 951us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.411002 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.411062 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.411243 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.411319 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:41:20.411966 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.401293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39071 (local address 127.1.53.65:33099)
0927 22:41:20.404845 (+ 3552us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:20.404861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:20.405038 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:20.405583 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:20.405587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:20.405598 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:41:20.406550 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.406562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.407734 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.407737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.409017 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:20.409028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.410394 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.410403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:20.410597 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.410673 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.411126 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.411133 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:41:20.411220 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:41:20.411367 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3339,"thread_start_us":3193,"threads_started":1}
I20260927 22:41:20.448632 8838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:20.448895 8838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:20.451606 8838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Sending DeleteTablet for 3 replicas of tablet ce8f4624964647f1a875820534506075
I20260927 22:41:20.452508 9113 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8f4624964647f1a875820534506075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44076
I20260927 22:41:20.452557 9257 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8f4624964647f1a875820534506075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54416
I20260927 22:41:20.452982 8974 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8f4624964647f1a875820534506075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44190
I20260927 22:41:20.461727 9418 tablet_replica.cc:330] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: stopping tablet replica
I20260927 22:41:20.462054 9418 raft_consensus.cc:2229] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:20.462430 9418 raft_consensus.cc:2258] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:20.465229 9418 ts_tablet_manager.cc:1905] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:20.472769 9417 tablet_replica.cc:330] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: stopping tablet replica
I20260927 22:41:20.473086 9418 ts_tablet_manager.cc:1918] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:41:20.473060 9417 raft_consensus.cc:2229] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:20.473294 9418 log.cc:1196] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/ce8f4624964647f1a875820534506075
I20260927 22:41:20.473385 9417 raft_consensus.cc:2258] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:20.474025 9416 tablet_replica.cc:330] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: stopping tablet replica
I20260927 22:41:20.474246 9416 raft_consensus.cc:2229] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:20.474603 9416 raft_consensus.cc:2258] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:20.475431 9417 ts_tablet_manager.cc:1905] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:20.480073 9417 ts_tablet_manager.cc:1918] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:41:20.480222 9417 log.cc:1196] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/ce8f4624964647f1a875820534506075
I20260927 22:41:20.480708 9417 ts_tablet_manager.cc:1939] T ce8f4624964647f1a875820534506075 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting consensus metadata
I20260927 22:41:20.481122 9416 ts_tablet_manager.cc:1905] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:20.482023 8825 catalog_manager.cc:4841] TS 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743): tablet ce8f4624964647f1a875820534506075 (table test-table [id=3e49996d06204a0bafcc974ac4863cf2]) successfully deleted
I20260927 22:41:20.484454 9418 ts_tablet_manager.cc:1939] T ce8f4624964647f1a875820534506075 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting consensus metadata
I20260927 22:41:20.485853 8825 catalog_manager.cc:4841] TS 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421): tablet ce8f4624964647f1a875820534506075 (table test-table [id=3e49996d06204a0bafcc974ac4863cf2]) successfully deleted
I20260927 22:41:20.487494 9416 ts_tablet_manager.cc:1918] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:41:20.487632 9416 log.cc:1196] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/ce8f4624964647f1a875820534506075
I20260927 22:41:20.488081 9416 ts_tablet_manager.cc:1939] T ce8f4624964647f1a875820534506075 P 56cb69e7e84a463d8d590281fbb198f7: Deleting consensus metadata
I20260927 22:41:20.489215 8822 catalog_manager.cc:4841] TS 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071): tablet ce8f4624964647f1a875820534506075 (table test-table [id=3e49996d06204a0bafcc974ac4863cf2]) successfully deleted
Sep 27 22:41:20 dist-test-slave-pp80 krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:41:20 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:20 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:41:20.524327 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:20.514239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.1.53.126:35005)
0927 22:41:20.514485 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.514494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.514830 (+ 336us) server_negotiation.cc:407] Connection header received
0927 22:41:20.515062 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.515066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.515130 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.515234 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:20.516584 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.517323 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.518668 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.519165 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.519307 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.523995 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58058: BlockingRecv error: recv got EOF from 127.0.0.1:58058 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:41:20 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM
I20260927 22:41:20.542922 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.527371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.1.53.126:35005)
0927 22:41:20.527541 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.527547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.527853 (+ 306us) server_negotiation.cc:407] Connection header received
0927 22:41:20.527985 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.527991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.528072 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.528181 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:20.529375 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.530479 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.531706 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.532394 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.532514 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.536462 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:20.536470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:20.536474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:20.536530 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:20.539943 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:20.540796 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:20.540810 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:20.540813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:20.540949 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:20.541349 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:20.541353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:20.541355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:20.541690 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:20.541961 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.542308 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.542541 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:41:20.546765 8838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:20.547489 8838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:20.554219 8974 tablet_service.cc:1467] Processing CreateTablet for tablet 88d1205c72384d9685023324eb411907 (DEFAULT_TABLE table=test-table [id=6a455f58d5964c639d6bbb8c142f1542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:20.554733 8974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d1205c72384d9685023324eb411907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.556900 9257 tablet_service.cc:1467] Processing CreateTablet for tablet 88d1205c72384d9685023324eb411907 (DEFAULT_TABLE table=test-table [id=6a455f58d5964c639d6bbb8c142f1542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:20.557358 9257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d1205c72384d9685023324eb411907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.557533 9113 tablet_service.cc:1467] Processing CreateTablet for tablet 88d1205c72384d9685023324eb411907 (DEFAULT_TABLE table=test-table [id=6a455f58d5964c639d6bbb8c142f1542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:20.557938 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d1205c72384d9685023324eb411907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.561687 9349 tablet_bootstrap.cc:492] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Bootstrap starting.
I20260927 22:41:20.563459 9349 tablet_bootstrap.cc:654] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.569056 9431 tablet_bootstrap.cc:492] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Bootstrap starting.
I20260927 22:41:20.571175 9431 tablet_bootstrap.cc:654] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.575186 9349 tablet_bootstrap.cc:492] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: No bootstrap required, opened a new log
I20260927 22:41:20.575345 9349 ts_tablet_manager.cc:1397] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 22:41:20.576095 9349 raft_consensus.cc:348] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.576370 9349 raft_consensus.cc:374] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.576462 9349 raft_consensus.cc:729] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.576697 9349 consensus_queue.cc:260] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.577891 9349 ts_tablet_manager.cc:1428] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:41:20.584014 9433 tablet_bootstrap.cc:492] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Bootstrap starting.
I20260927 22:41:20.585559 9431 tablet_bootstrap.cc:492] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: No bootstrap required, opened a new log
I20260927 22:41:20.585870 9431 ts_tablet_manager.cc:1397] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260927 22:41:20.586593 9431 raft_consensus.cc:348] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.586772 9433 tablet_bootstrap.cc:654] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.586805 9431 raft_consensus.cc:374] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.586894 9431 raft_consensus.cc:729] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.587157 9431 consensus_queue.cc:260] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.587759 9431 ts_tablet_manager.cc:1428] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:41:20.589359 9433 tablet_bootstrap.cc:492] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: No bootstrap required, opened a new log
I20260927 22:41:20.589697 9433 ts_tablet_manager.cc:1397] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 22:41:20.590488 9433 raft_consensus.cc:348] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.590758 9433 raft_consensus.cc:374] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.590853 9433 raft_consensus.cc:729] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b0835765d8474e9f66bb7ae4e0b80d, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.591121 9433 consensus_queue.cc:260] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.591777 9433 ts_tablet_manager.cc:1428] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:41:20.742849 9404 raft_consensus.cc:482] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:20.743112 9404 raft_consensus.cc:504] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.743939 9404 leader_election.cc:288] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743), 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:20.744613 9278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d1205c72384d9685023324eb411907" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cb69e7e84a463d8d590281fbb198f7" is_pre_election: true
I20260927 22:41:20.744853 9278 raft_consensus.cc:2454] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 0.
I20260927 22:41:20.745312 8925 leader_election.cc:302] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543a1928c07483f9ae1ee7c0b24b527, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:20.745505 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d1205c72384d9685023324eb411907" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" is_pre_election: true
I20260927 22:41:20.745677 9404 raft_consensus.cc:2781] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:20.745795 9404 raft_consensus.cc:482] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:20.745779 9133 raft_consensus.cc:2454] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 0.
I20260927 22:41:20.745886 9404 raft_consensus.cc:3037] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:20.749049 9404 raft_consensus.cc:504] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.749696 9404 leader_election.cc:288] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 election: Requested vote from peers 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743), 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:20.750260 9278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d1205c72384d9685023324eb411907" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cb69e7e84a463d8d590281fbb198f7"
I20260927 22:41:20.750284 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d1205c72384d9685023324eb411907" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
I20260927 22:41:20.750464 9278 raft_consensus.cc:3037] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:20.750512 9133 raft_consensus.cc:3037] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:20.752559 9133 raft_consensus.cc:2454] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 1.
I20260927 22:41:20.752954 8928 leader_election.cc:302] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b0835765d8474e9f66bb7ae4e0b80d, 2543a1928c07483f9ae1ee7c0b24b527; no voters:
I20260927 22:41:20.753248 9404 raft_consensus.cc:2781] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:20.753454 9404 raft_consensus.cc:686] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 LEADER]: Becoming Leader. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Running, Role: LEADER
I20260927 22:41:20.753486 9278 raft_consensus.cc:2454] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 1.
I20260927 22:41:20.753731 9404 consensus_queue.cc:237] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.757205 8837 catalog_manager.cc:5495] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } health_report { overall_health: UNKNOWN } } }
I20260927 22:41:20.816524 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:20.811579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.1.53.65:35421)
0927 22:41:20.811791 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:20.811797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:20.811895 (+ 98us) server_negotiation.cc:407] Connection header received
0927 22:41:20.812173 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:20.812178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:20.812290 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:20.812407 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:20.812750 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.813420 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.814750 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:20.815159 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:20.815291 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:20.815732 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:20.815860 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:20.815954 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:20.816122 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:41:20.816199 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:41:20.819641 9133 raft_consensus.cc:1261] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Refusing update from remote peer 2543a1928c07483f9ae1ee7c0b24b527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:20.819777 9278 raft_consensus.cc:1261] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Refusing update from remote peer 2543a1928c07483f9ae1ee7c0b24b527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:20.820343 9404 consensus_queue.cc:1035] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:20.820611 9407 consensus_queue.cc:1035] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:20.827111 9399 mvcc.cc:205] Tried to move back new op lower bound from 7334088215832117248 to 7334088215570386944. Current Snapshot: MvccSnapshot[applied={T|T < 7334088215832117248}]
I20260927 22:41:20.840083 8838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:20.840408 8838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:20.844811 8838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Sending DeleteTablet for 3 replicas of tablet 88d1205c72384d9685023324eb411907
I20260927 22:41:20.845925 9257 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d1205c72384d9685023324eb411907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54416
I20260927 22:41:20.846266 9416 tablet_replica.cc:330] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: stopping tablet replica
I20260927 22:41:20.846294 8974 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d1205c72384d9685023324eb411907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44190
I20260927 22:41:20.846289 9114 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d1205c72384d9685023324eb411907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44076
I20260927 22:41:20.846472 9416 raft_consensus.cc:2229] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:20.846570 9113 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d1205c72384d9685023324eb411907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:20 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44076
I20260927 22:41:20.846714 9416 raft_consensus.cc:2258] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:20.846863 9418 tablet_replica.cc:330] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: stopping tablet replica
I20260927 22:41:20.847095 9418 raft_consensus.cc:2229] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:20.847605 9418 raft_consensus.cc:2258] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:20.846581 9417 tablet_replica.cc:330] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: stopping tablet replica
I20260927 22:41:20.847955 9417 raft_consensus.cc:2229] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:20.848189 9417 raft_consensus.cc:2258] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:20.848839 9416 ts_tablet_manager.cc:1905] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:20.849081 9417 ts_tablet_manager.cc:1905] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:20.849081 9418 ts_tablet_manager.cc:1905] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:20.853181 8838 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:20.853883 8838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:20.855908 9418 ts_tablet_manager.cc:1918] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:20.855904 9417 ts_tablet_manager.cc:1918] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:20.856067 9418 log.cc:1196] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/88d1205c72384d9685023324eb411907
I20260927 22:41:20.856067 9417 log.cc:1196] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/88d1205c72384d9685023324eb411907
I20260927 22:41:20.856494 9417 ts_tablet_manager.cc:1939] T 88d1205c72384d9685023324eb411907 P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting consensus metadata
I20260927 22:41:20.856510 9418 ts_tablet_manager.cc:1939] T 88d1205c72384d9685023324eb411907 P 2543a1928c07483f9ae1ee7c0b24b527: Deleting consensus metadata
I20260927 22:41:20.857746 8825 catalog_manager.cc:4841] TS 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421): tablet 88d1205c72384d9685023324eb411907 (table test-table [id=6a455f58d5964c639d6bbb8c142f1542]) successfully deleted
I20260927 22:41:20.857988 8825 catalog_manager.cc:4841] TS 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743): tablet 88d1205c72384d9685023324eb411907 (table test-table [id=6a455f58d5964c639d6bbb8c142f1542]) successfully deleted
W20260927 22:41:20.858374 8825 catalog_manager.cc:4808] TS 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743): delete failed for tablet 88d1205c72384d9685023324eb411907 because the tablet was not found. No further retry: Not found: Tablet not found: 88d1205c72384d9685023324eb411907
I20260927 22:41:20.860576 9416 ts_tablet_manager.cc:1918] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:20.860749 9416 log.cc:1196] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/88d1205c72384d9685023324eb411907
I20260927 22:41:20.861102 9416 ts_tablet_manager.cc:1939] T 88d1205c72384d9685023324eb411907 P 56cb69e7e84a463d8d590281fbb198f7: Deleting consensus metadata
I20260927 22:41:20.862099 8822 catalog_manager.cc:4841] TS 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071): tablet 88d1205c72384d9685023324eb411907 (table test-table [id=6a455f58d5964c639d6bbb8c142f1542]) successfully deleted
I20260927 22:41:20.862637 8974 tablet_service.cc:1467] Processing CreateTablet for tablet fe528a6d51cd42ca8e05fe228f0f7c5e (DEFAULT_TABLE table=test-table [id=ac0dfef430bc4faf955eaa882e14d3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:20.862898 9257 tablet_service.cc:1467] Processing CreateTablet for tablet fe528a6d51cd42ca8e05fe228f0f7c5e (DEFAULT_TABLE table=test-table [id=ac0dfef430bc4faf955eaa882e14d3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:20.863054 8974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe528a6d51cd42ca8e05fe228f0f7c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.863291 9257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe528a6d51cd42ca8e05fe228f0f7c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.863979 9113 tablet_service.cc:1467] Processing CreateTablet for tablet fe528a6d51cd42ca8e05fe228f0f7c5e (DEFAULT_TABLE table=test-table [id=ac0dfef430bc4faf955eaa882e14d3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:20.864374 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe528a6d51cd42ca8e05fe228f0f7c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:20.869455 9431 tablet_bootstrap.cc:492] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Bootstrap starting.
I20260927 22:41:20.870608 9349 tablet_bootstrap.cc:492] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Bootstrap starting.
I20260927 22:41:20.871033 9431 tablet_bootstrap.cc:654] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.872354 9349 tablet_bootstrap.cc:654] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.874605 9433 tablet_bootstrap.cc:492] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Bootstrap starting.
I20260927 22:41:20.875113 9431 tablet_bootstrap.cc:492] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: No bootstrap required, opened a new log
I20260927 22:41:20.875340 9431 ts_tablet_manager.cc:1397] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260927 22:41:20.875665 9349 tablet_bootstrap.cc:492] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: No bootstrap required, opened a new log
I20260927 22:41:20.875834 9349 ts_tablet_manager.cc:1397] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260927 22:41:20.876407 9433 tablet_bootstrap.cc:654] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:20.876365 9431 raft_consensus.cc:348] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.876606 9349 raft_consensus.cc:348] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.876873 9349 raft_consensus.cc:374] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.876804 9431 raft_consensus.cc:374] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.876962 9349 raft_consensus.cc:729] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.877089 9431 raft_consensus.cc:729] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb69e7e84a463d8d590281fbb198f7, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.877401 9349 consensus_queue.cc:260] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.877940 9349 ts_tablet_manager.cc:1428] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:41:20.877456 9431 consensus_queue.cc:260] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.880261 9431 ts_tablet_manager.cc:1428] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:41:20.884877 9433 tablet_bootstrap.cc:492] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: No bootstrap required, opened a new log
I20260927 22:41:20.885051 9433 ts_tablet_manager.cc:1397] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260927 22:41:20.885931 9433 raft_consensus.cc:348] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.886227 9433 raft_consensus.cc:374] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:20.886314 9433 raft_consensus.cc:729] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b0835765d8474e9f66bb7ae4e0b80d, State: Initialized, Role: FOLLOWER
I20260927 22:41:20.886600 9433 consensus_queue.cc:260] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:20.887152 9433 ts_tablet_manager.cc:1428] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:41:21.095839 9404 raft_consensus.cc:482] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:21.096067 9404 raft_consensus.cc:504] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:21.096942 9404 leader_election.cc:288] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743), 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:21.097544 9277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe528a6d51cd42ca8e05fe228f0f7c5e" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cb69e7e84a463d8d590281fbb198f7" is_pre_election: true
I20260927 22:41:21.097563 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe528a6d51cd42ca8e05fe228f0f7c5e" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" is_pre_election: true
I20260927 22:41:21.097796 9277 raft_consensus.cc:2454] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 0.
I20260927 22:41:21.097808 9133 raft_consensus.cc:2454] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 0.
I20260927 22:41:21.098266 8925 leader_election.cc:302] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543a1928c07483f9ae1ee7c0b24b527, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:21.098551 9404 raft_consensus.cc:2781] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:21.098709 9404 raft_consensus.cc:482] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:21.098837 9404 raft_consensus.cc:3037] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:21.101125 9404 raft_consensus.cc:504] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:21.101701 9404 leader_election.cc:288] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 election: Requested vote from peers 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743), 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071)
I20260927 22:41:21.102221 9277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe528a6d51cd42ca8e05fe228f0f7c5e" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cb69e7e84a463d8d590281fbb198f7"
I20260927 22:41:21.102279 9133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe528a6d51cd42ca8e05fe228f0f7c5e" candidate_uuid: "2543a1928c07483f9ae1ee7c0b24b527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b0835765d8474e9f66bb7ae4e0b80d"
I20260927 22:41:21.102428 9277 raft_consensus.cc:3037] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:21.102521 9133 raft_consensus.cc:3037] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:21.104987 9277 raft_consensus.cc:2454] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 1.
I20260927 22:41:21.105540 9133 raft_consensus.cc:2454] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543a1928c07483f9ae1ee7c0b24b527 in term 1.
I20260927 22:41:21.105683 8925 leader_election.cc:302] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543a1928c07483f9ae1ee7c0b24b527, 56cb69e7e84a463d8d590281fbb198f7; no voters:
I20260927 22:41:21.106034 9404 raft_consensus.cc:2781] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:21.106201 9404 raft_consensus.cc:686] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 LEADER]: Becoming Leader. State: Replica: 2543a1928c07483f9ae1ee7c0b24b527, State: Running, Role: LEADER
I20260927 22:41:21.106496 9404 consensus_queue.cc:237] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } }
I20260927 22:41:21.109438 8837 catalog_manager.cc:5495] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "2543a1928c07483f9ae1ee7c0b24b527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2543a1928c07483f9ae1ee7c0b24b527" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 35421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 } health_report { overall_health: UNKNOWN } } }
I20260927 22:41:21.145292 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:21.138623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:41911 (local address 127.1.53.65:35421)
0927 22:41:21.138848 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:21.138854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:21.139449 (+ 595us) server_negotiation.cc:407] Connection header received
0927 22:41:21.139596 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:21.139604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:21.139690 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:21.139800 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:21.140598 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.141773 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.143191 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.143949 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.144099 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:21.144524 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:21.144665 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:21.144764 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:21.144851 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 22:41:21.144953 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":61}
I20260927 22:41:21.145325 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:21.138635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:35421 (local address 127.1.53.65:41911)
0927 22:41:21.139346 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:21.139364 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:21.139468 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:21.139845 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:21.139850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:21.139864 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:41:21.140390 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:21.140402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.141926 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.141930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:21.143047 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:21.143058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.144135 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.144139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:21.144311 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:21.144358 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:41:21.144759 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:41:21.144767 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:41:21.144825 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 22:41:21.144986 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":3,"thread_start_us":438,"threads_started":1}
I20260927 22:41:21.153637 9134 raft_consensus.cc:1261] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Refusing update from remote peer 2543a1928c07483f9ae1ee7c0b24b527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:21.153703 9277 raft_consensus.cc:1261] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Refusing update from remote peer 2543a1928c07483f9ae1ee7c0b24b527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:21.154249 9353 consensus_queue.cc:1035] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b0835765d8474e9f66bb7ae4e0b80d" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:21.154780 9407 consensus_queue.cc:1035] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cb69e7e84a463d8d590281fbb198f7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:21.218791 8837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:21.219086 8837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:21.222405 8837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1374a0ecdbd74477b19d6de93b73e842: Sending DeleteTablet for 3 replicas of tablet fe528a6d51cd42ca8e05fe228f0f7c5e
I20260927 22:41:21.223327 9113 tablet_service.cc:1514] Processing DeleteTablet for tablet fe528a6d51cd42ca8e05fe228f0f7c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:21 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44076
I20260927 22:41:21.223439 9257 tablet_service.cc:1514] Processing DeleteTablet for tablet fe528a6d51cd42ca8e05fe228f0f7c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:21 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:54416
I20260927 22:41:21.223641 9417 tablet_replica.cc:330] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: stopping tablet replica
I20260927 22:41:21.223757 9416 tablet_replica.cc:330] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: stopping tablet replica
I20260927 22:41:21.223824 9417 raft_consensus.cc:2229] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:21.223932 9416 raft_consensus.cc:2229] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:21.224033 9417 raft_consensus.cc:2258] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:21.224138 9416 raft_consensus.cc:2258] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:21.224687 8974 tablet_service.cc:1514] Processing DeleteTablet for tablet fe528a6d51cd42ca8e05fe228f0f7c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:21 UTC) from {username='oryx', principal='oryx/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44190
I20260927 22:41:21.225000 9418 tablet_replica.cc:330] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: stopping tablet replica
I20260927 22:41:21.225175 9418 raft_consensus.cc:2229] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:21.225227 9417 ts_tablet_manager.cc:1905] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:21.225518 9418 raft_consensus.cc:2258] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:21.225590 9416 ts_tablet_manager.cc:1905] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:21.226965 9418 ts_tablet_manager.cc:1905] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:21.232297 9417 ts_tablet_manager.cc:1918] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:41:21.232336 9416 ts_tablet_manager.cc:1918] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:41:21.232446 9417 log.cc:1196] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/fe528a6d51cd42ca8e05fe228f0f7c5e
I20260927 22:41:21.232460 9416 log.cc:1196] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/fe528a6d51cd42ca8e05fe228f0f7c5e
I20260927 22:41:21.232600 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 8909
I20260927 22:41:21.232884 9416 ts_tablet_manager.cc:1939] T fe528a6d51cd42ca8e05fe228f0f7c5e P 56cb69e7e84a463d8d590281fbb198f7: Deleting consensus metadata
I20260927 22:41:21.232890 9417 ts_tablet_manager.cc:1939] T fe528a6d51cd42ca8e05fe228f0f7c5e P 18b0835765d8474e9f66bb7ae4e0b80d: Deleting consensus metadata
I20260927 22:41:21.234232 8825 catalog_manager.cc:4841] TS 18b0835765d8474e9f66bb7ae4e0b80d (127.1.53.66:35743): tablet fe528a6d51cd42ca8e05fe228f0f7c5e (table test-table [id=ac0dfef430bc4faf955eaa882e14d3b3]) successfully deleted
I20260927 22:41:21.234912 8822 catalog_manager.cc:4841] TS 56cb69e7e84a463d8d590281fbb198f7 (127.1.53.67:39071): tablet fe528a6d51cd42ca8e05fe228f0f7c5e (table test-table [id=ac0dfef430bc4faf955eaa882e14d3b3]) successfully deleted
I20260927 22:41:21.238063 9418 ts_tablet_manager.cc:1918] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:41:21.238215 9418 log.cc:1196] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/fe528a6d51cd42ca8e05fe228f0f7c5e
I20260927 22:41:21.238719 9418 ts_tablet_manager.cc:1939] T fe528a6d51cd42ca8e05fe228f0f7c5e P 2543a1928c07483f9ae1ee7c0b24b527: Deleting consensus metadata
I20260927 22:41:21.239951 8825 catalog_manager.cc:4841] TS 2543a1928c07483f9ae1ee7c0b24b527 (127.1.53.65:35421): tablet fe528a6d51cd42ca8e05fe228f0f7c5e (table test-table [id=ac0dfef430bc4faf955eaa882e14d3b3]) successfully deleted
Sep 27 22:41:21 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.65@KRBTEST.COM
I20260927 22:41:21.253988 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:21.233468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.1.53.65:35421)
0927 22:41:21.233629 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:21.233634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:21.238877 (+ 5243us) server_negotiation.cc:407] Connection header received
0927 22:41:21.239116 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:21.239125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:21.239269 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:21.239391 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:21.240947 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.241819 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.243026 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.243466 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.243605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:21.247785 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:21.247799 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:21.247815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:21.247853 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:21.251047 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:21.251820 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:21.251827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:21.251830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:21.251983 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:21.252409 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:21.252412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:21.252414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:21.252734 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:21.253062 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:21.253332 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:41:21.253566 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:41:21.254717 9040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:21.412958 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 8909
I20260927 22:41:21.433789 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 9049
Sep 27 22:41:21 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.66@KRBTEST.COM
I20260927 22:41:21.449055 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:21.434571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.1.53.66:35743)
0927 22:41:21.435022 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:21.435026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:21.435043 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:41:21.435096 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:21.435100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:21.435166 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:21.435263 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:21.436510 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.437260 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.438448 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.438849 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.438948 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:21.443593 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:21.443606 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:21.443622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:21.443663 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:21.446500 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:21.447183 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:21.447190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:21.447193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:21.447327 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:21.447714 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:21.447718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:21.447720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:21.448075 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:21.448387 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:21.448622 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:41:21.448777 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260927 22:41:21.449702 9179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:21.545706 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 9049
I20260927 22:41:21.564639 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 9188
Sep 27 22:41:21 dist-test-slave-pp80 krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.53.67@KRBTEST.COM
I20260927 22:41:21.580873 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:21.565496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.1.53.67:39071)
0927 22:41:21.566056 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:21.566060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:21.566077 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:41:21.566128 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:21.566132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:21.566197 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:21.566289 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:21.567633 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.568491 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.569915 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:21.570439 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:21.570533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:21.575100 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:21.575115 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:21.575130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:21.575161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:21.578093 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:21.578781 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:21.578789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:21.578791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:21.578985 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:21.579382 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:21.579386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:21.579388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:21.579724 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:21.579930 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:21.580298 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:41:21.580487 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":330,"threads_started":1}
I20260927 22:41:21.581578 9318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:21.706988 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 9188
I20260927 22:41:21.728458 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 8806
I20260927 22:41:21.729641 8878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:21.843024 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 8806
2026-09-27T22:41:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6029 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:21 dist-test-slave-pp80 krb5kdc[9460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:21 dist-test-slave-pp80 krb5kdc[9460](info): set up 2 sockets
Sep 27 22:41:21 dist-test-slave-pp80 krb5kdc[9460](info): commencing operation
krb5kdc: starting...
W20260927 22:41:22.949553 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:23 dist-test-slave-pp80 krb5kdc[9460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:41:23.171993 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46651
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:23.488484 9473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:23.488766 9473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:23.488932 9473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:23.496090 9473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:23.496183 9473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:23.496276 9473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:23.496366 9473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:23.496461 9473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:23.505777 9473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:23.507079 9473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:23.508363 9473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:23.520161 9478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:23.520138 9481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:23.520097 9479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:23.525974 9473 server_base.cc:1034] running on GCE node
I20260927 22:41:23.526835 9473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:23.528453 9473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:23.529747 9473 hybrid_clock.cc:648] HybridClock initialized: now 1790548883529653 us; error 54 us; skew 500 ppm
I20260927 22:41:23.530115 9473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:41:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1764 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:23 dist-test-slave-pp80 krb5kdc[9487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:23 dist-test-slave-pp80 krb5kdc[9487](info): set up 2 sockets
Sep 27 22:41:23 dist-test-slave-pp80 krb5kdc[9487](info): commencing operation
krb5kdc: starting...
W20260927 22:41:24.761221 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:24 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:41:24.984601 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45165
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:45165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:25.319496 9500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:25.319805 9500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:25.319984 9500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:25.327858 9500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:25.327970 9500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:25.328016 9500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:25.328131 9500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:25.328192 9500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:25.337991 9500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:45165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:25.339323 9500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:25.340667 9500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:25.352274 9505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:25.352346 9508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:25.352305 9506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:25.353116 9500 server_base.cc:1034] running on GCE node
I20260927 22:41:25.353858 9500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:25.355259 9500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:25.356495 9500 hybrid_clock.cc:648] HybridClock initialized: now 1790548885356452 us; error 53 us; skew 500 ppm
I20260927 22:41:25.356866 9500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:25 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548885, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:25.360092 9500 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:41:25.361620 9500 webserver.cc:460] Webserver started at http://127.1.53.126:46357/ using document root <none> and password file <none>
I20260927 22:41:25.362327 9500 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:25.362473 9500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:25.362790 9500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:25.365772 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "8ae6ff051854418e9edc8209ae27f5bd"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "aec8b946f87554e7da8ff3fb0f840a99"
server_key_iv: "4bad4a1c2c4ba11f40389635b6196119"
server_key_version: "encryptionkey@0"
I20260927 22:41:25.366607 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "8ae6ff051854418e9edc8209ae27f5bd"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "aec8b946f87554e7da8ff3fb0f840a99"
server_key_iv: "4bad4a1c2c4ba11f40389635b6196119"
server_key_version: "encryptionkey@0"
I20260927 22:41:25.371949 9500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:41:25.375711 9515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:25.377256 9500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:25.377453 9500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "8ae6ff051854418e9edc8209ae27f5bd"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "aec8b946f87554e7da8ff3fb0f840a99"
server_key_iv: "4bad4a1c2c4ba11f40389635b6196119"
server_key_version: "encryptionkey@0"
I20260927 22:41:25.377657 9500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:25.407270 9500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:25.411891 9500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:25.412209 9500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:25.430945 9500 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:45165
I20260927 22:41:25.431034 9566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:45165 every 8 connection(s)
I20260927 22:41:25.432355 9500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:25.436121 9567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:25.436448 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 9500
I20260927 22:41:25.436655 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:25.437494 1237 external_mini_cluster.cc:1428] Setting key 84e2936cd25f7ecdf0a5d9d125ae20b3
I20260927 22:41:25.450709 9567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Bootstrap starting.
Sep 27 22:41:25 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:25.454299 9567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:25.455669 9567 log.cc:824] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:25.458858 9567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: No bootstrap required, opened a new log
I20260927 22:41:25.461445 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:25.439542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.1.53.126:45165)
0927 22:41:25.440347 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:25.440369 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:25.440417 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:41:25.442033 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:25.442065 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:25.442555 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:25.443004 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:25.444324 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:25.446198 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:25.447627 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:25.448733 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:25.448936 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:25.453296 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:25.453306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:25.453327 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:25.453381 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:25.457064 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:25.457767 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:25.457774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:25.457784 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:25.457997 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:25.458356 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:25.458361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:25.458364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:25.459023 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:25.459289 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:25.459694 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 22:41:25.460098 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":329,"threads_started":1}
I20260927 22:41:25.467468 9567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:25.467782 9567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:25.467846 9567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae6ff051854418e9edc8209ae27f5bd, State: Initialized, Role: FOLLOWER
I20260927 22:41:25.468416 9567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:25.468617 9567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:25.468740 9567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:25.468915 9567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:25.471149 9567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:25.471578 9567 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae6ff051854418e9edc8209ae27f5bd; no voters:
I20260927 22:41:25.472069 9567 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:25.472213 9572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:25.472972 9572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 LEADER]: Becoming Leader. State: Replica: 8ae6ff051854418e9edc8209ae27f5bd, State: Running, Role: LEADER
I20260927 22:41:25.473485 9572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:25.474061 9567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:25.476289 9573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ae6ff051854418e9edc8209ae27f5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } } }
I20260927 22:41:25.476321 9574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae6ff051854418e9edc8209ae27f5bd. Latest consensus state: current_term: 1 leader_uuid: "8ae6ff051854418e9edc8209ae27f5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } } }
I20260927 22:41:25.476841 9573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:25.476898 9574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:25.477946 9578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:25.483654 9578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:25.491973 9578 catalog_manager.cc:1347] Generated new cluster ID: 05521c2376d74e00bf8be073d80aea4b
I20260927 22:41:25.492094 9578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:25.509518 9578 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:25.510843 9578 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:25.522758 9578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Generated new TSK 0
I20260927 22:41:25.523597 9578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:41:25.584374 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45165
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:25.914925 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:25.915200 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:25.915294 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:25.922364 9593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:25.922467 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:25.922580 9593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:25.931902 9593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:25.933084 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:25.934415 9593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:25.946915 9601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:25.946877 9599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:25.947119 9598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:25.947829 9593 server_base.cc:1034] running on GCE node
I20260927 22:41:25.948495 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:25.949328 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:25.950542 9593 hybrid_clock.cc:648] HybridClock initialized: now 1790548885950499 us; error 34 us; skew 500 ppm
I20260927 22:41:25.950870 9593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:25 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548885, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:25.954137 9593 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:41:25.955482 9593 webserver.cc:460] Webserver started at http://127.1.53.65:42225/ using document root <none> and password file <none>
I20260927 22:41:25.956133 9593 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:25.956280 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:25.956598 9593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:25.959362 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "2cede6de25cd4387b64659406c7fa8b5"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "48148e090dd199781fb9c780f67dc045"
server_key_iv: "918d3076ceba736e4a41ebded81011c2"
server_key_version: "encryptionkey@0"
I20260927 22:41:25.960119 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "2cede6de25cd4387b64659406c7fa8b5"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "48148e090dd199781fb9c780f67dc045"
server_key_iv: "918d3076ceba736e4a41ebded81011c2"
server_key_version: "encryptionkey@0"
I20260927 22:41:25.965162 9593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:41:25.968477 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:25.969774 9593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:41:25.969934 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "2cede6de25cd4387b64659406c7fa8b5"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "48148e090dd199781fb9c780f67dc045"
server_key_iv: "918d3076ceba736e4a41ebded81011c2"
server_key_version: "encryptionkey@0"
I20260927 22:41:25.970116 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:25.988559 9593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:25.992775 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:25.993074 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:25.994105 9593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:25.995566 9593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:25.995637 9593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:25.995761 9593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:25.995823 9593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:26.029625 9593 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:39517
I20260927 22:41:26.029672 9720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:39517 every 8 connection(s)
I20260927 22:41:26.030932 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:26.035915 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 9593
I20260927 22:41:26.036116 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:26.036803 1237 external_mini_cluster.cc:1428] Setting key 623ea42327fbb3523593edaadc57ea6f
Sep 27 22:41:26 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548885, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:26.055262 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:26.033483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:34159 (local address 127.1.53.126:45165)
0927 22:41:26.033954 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:26.033959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:26.035019 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 22:41:26.037183 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:26.037189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:26.037274 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:26.037406 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:26.039130 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.040327 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.042805 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.043525 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.043679 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:26.048108 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:26.048117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:26.048125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:26.048169 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:26.051805 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:26.052648 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:26.052656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:26.052663 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:26.052811 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:26.053222 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:26.053230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:26.053236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:26.053624 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:26.053776 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:26.054519 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 22:41:26.054784 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260927 22:41:26.055970 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:26.034070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:34159)
0927 22:41:26.034821 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:26.034886 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:26.036852 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:26.037723 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:26.037732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:26.038271 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:26.038961 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:26.038978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.040506 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.040511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:26.042636 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:26.042653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.043767 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.043776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:26.044097 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:26.044957 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:26.044997 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:26.047858 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:26.051975 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:26.051983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:26.051996 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:26.052523 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:26.052925 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:26.052928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:26.052930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:26.053104 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:26.053775 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:26.053782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:26.054116 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 22:41:26.054371 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:41:26.054766 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":308,"threads_started":1}
I20260927 22:41:26.057667 9721 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45165
I20260927 22:41:26.057978 9721 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:26.058709 9721 heartbeater.cc:511] Master 127.1.53.126:45165 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:41:26.061213 9532 ts_manager.cc:194] Registered new tserver with Master: 2cede6de25cd4387b64659406c7fa8b5 (127.1.53.65:39517)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:41:26.063032 9532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:34159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:41:26.091094 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45165
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:26.417375 9727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:26.417654 9727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:26.417805 9727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:26.425151 9727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:26.425261 9727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:26.425441 9727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:26.435102 9727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:26.436388 9727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:26.437650 9727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:26.450742 9733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:26.450799 9732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:26.450714 9735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:26.451407 9727 server_base.cc:1034] running on GCE node
I20260927 22:41:26.452011 9727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:26.453022 9727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:26.454244 9727 hybrid_clock.cc:648] HybridClock initialized: now 1790548886454191 us; error 52 us; skew 500 ppm
I20260927 22:41:26.454556 9727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:26 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:26.457631 9727 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:41:26.459052 9727 webserver.cc:460] Webserver started at http://127.1.53.66:34133/ using document root <none> and password file <none>
I20260927 22:41:26.459707 9727 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:26.459851 9727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:26.460150 9727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:26.462921 9727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "92fe692a041f4e00aa5ccfbd7da6daad"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "640ddf767ce3b9c22976c3977299d3b1"
server_key_iv: "e92146776e25c9e206cdfea3876fec7a"
server_key_version: "encryptionkey@0"
I20260927 22:41:26.463671 9727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "92fe692a041f4e00aa5ccfbd7da6daad"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "640ddf767ce3b9c22976c3977299d3b1"
server_key_iv: "e92146776e25c9e206cdfea3876fec7a"
server_key_version: "encryptionkey@0"
I20260927 22:41:26.468688 9727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:41:26.472036 9742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:26.473555 9727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:26.473721 9727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "92fe692a041f4e00aa5ccfbd7da6daad"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "640ddf767ce3b9c22976c3977299d3b1"
server_key_iv: "e92146776e25c9e206cdfea3876fec7a"
server_key_version: "encryptionkey@0"
I20260927 22:41:26.473906 9727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:26.496218 9727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:26.500464 9727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:26.500759 9727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:26.501751 9727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:26.503100 9727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:26.503201 9727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:26.503340 9727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:26.503410 9727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:26.537252 9727 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:33177
I20260927 22:41:26.537350 9854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:33177 every 8 connection(s)
I20260927 22:41:26.538549 9727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:26.542769 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 9727
I20260927 22:41:26.542963 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:26.543392 1237 external_mini_cluster.cc:1428] Setting key 4e27f55c56c993e8035ce9bd58b3f99b
Sep 27 22:41:26 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:26.562428 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:26.541158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:52999 (local address 127.1.53.126:45165)
0927 22:41:26.541376 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:26.541381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:26.542747 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 22:41:26.545301 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:26.545308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:26.545400 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:26.545477 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:26.547153 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.548329 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.550238 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.550993 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.551135 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:26.554744 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:26.554754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:26.554763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:26.554811 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:26.558715 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:26.559601 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:26.559611 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:26.559618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:26.559781 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:26.560289 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:26.560297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:26.560304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:26.560742 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:26.560903 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:26.561626 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 22:41:26.561922 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:41:26.563177 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:26.541664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:52999)
0927 22:41:26.542580 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:26.542621 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:26.544992 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:26.545779 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:26.545791 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:26.546365 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:26.546967 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:26.546985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.548491 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.548495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:26.550086 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:26.550099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:26.551221 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:26.551227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:26.551454 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:26.552288 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:26.552335 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:26.554523 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:26.558904 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:26.558913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:26.558928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:26.559460 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:26.559932 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:26.559936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:26.559938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:26.560132 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:26.560892 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:26.560898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:26.561239 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 22:41:26.561479 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 22:41:26.561898 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":289,"threads_started":1}
I20260927 22:41:26.564790 9855 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45165
I20260927 22:41:26.565203 9855 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:26.566243 9855 heartbeater.cc:511] Master 127.1.53.126:45165 requested a full tablet report, sending...
I20260927 22:41:26.568042 9532 ts_manager.cc:194] Registered new tserver with Master: 92fe692a041f4e00aa5ccfbd7da6daad (127.1.53.66:33177)
I20260927 22:41:26.568910 9532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:52999
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:41:26.600771 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45165
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:26.945358 9860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:26.945637 9860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:26.945828 9860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:26.953128 9860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:26.953233 9860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:26.953452 9860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:26.963065 9860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:26.964322 9860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:26.965571 9860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:26.978574 9866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:26.978678 9868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:26.978753 9865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:26.979739 9860 server_base.cc:1034] running on GCE node
I20260927 22:41:26.980338 9860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:26.981403 9860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:26.982599 9860 hybrid_clock.cc:648] HybridClock initialized: now 1790548886982540 us; error 48 us; skew 500 ppm
I20260927 22:41:26.982944 9860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:26 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:26.986294 9860 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:41:26.987735 9860 webserver.cc:460] Webserver started at http://127.1.53.67:45187/ using document root <none> and password file <none>
I20260927 22:41:26.988505 9860 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:26.988634 9860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:26.988967 9860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:26.991832 9860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "361e980dd4dc478eb5a2ccaf310d9fdc"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "6940b84aa04c1c79b748c973eaa6f586"
server_key_iv: "be49f03430b4f45d18a5daf6b8341951"
server_key_version: "encryptionkey@0"
I20260927 22:41:26.992704 9860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "361e980dd4dc478eb5a2ccaf310d9fdc"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "6940b84aa04c1c79b748c973eaa6f586"
server_key_iv: "be49f03430b4f45d18a5daf6b8341951"
server_key_version: "encryptionkey@0"
I20260927 22:41:26.997754 9860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:41:27.001359 9875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:27.002745 9860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:41:27.002949 9860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "361e980dd4dc478eb5a2ccaf310d9fdc"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "6940b84aa04c1c79b748c973eaa6f586"
server_key_iv: "be49f03430b4f45d18a5daf6b8341951"
server_key_version: "encryptionkey@0"
I20260927 22:41:27.003160 9860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:27.016557 9860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:27.020658 9860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:27.020946 9860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:27.021845 9860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:27.023109 9860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:27.023211 9860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:27.023360 9860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:27.023437 9860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:27.057072 9860 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:42681
I20260927 22:41:27.057157 9987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:42681 every 8 connection(s)
I20260927 22:41:27.058434 9860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:27.063344 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 9860
I20260927 22:41:27.063544 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:27.064132 1237 external_mini_cluster.cc:1428] Setting key 436a92608a6636539d62e359c08cdfac
I20260927 22:41:27.066344 9721 heartbeater.cc:503] Master 127.1.53.126:45165 was elected leader, sending a full tablet report...
Sep 27 22:41:27 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:27.080533 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:27.061403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:48497 (local address 127.1.53.126:45165)
0927 22:41:27.061619 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:27.061625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:27.062736 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 22:41:27.065330 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:27.065335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:27.065421 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:27.065538 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:27.067652 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.068496 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.070028 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.070429 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.070555 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:27.074269 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:27.074275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:27.074278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:27.074311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:27.077121 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.077876 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.077881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.077884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.078050 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.078623 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.078627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.078629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.079005 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:27.079111 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:27.080014 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 22:41:27.080224 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:41:27.081763 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:27.061790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:48497)
0927 22:41:27.062556 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:27.062599 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:27.065034 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:27.065843 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:27.065913 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:27.066670 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:27.067493 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:27.067512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.068643 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.068647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:27.069886 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:27.069897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.070656 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.070663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:27.070904 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:27.071595 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:27.071617 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:27.074050 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:27.077328 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:27.077335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:27.077349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:27.077765 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:27.078207 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:27.078213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:27.078217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:27.078500 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:27.079134 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:27.079143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:27.079537 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 22:41:27.079910 (+ 373us) client_negotiation.cc:240] Negotiation successful
0927 22:41:27.080491 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":309,"threads_started":1}
I20260927 22:41:27.083305 9988 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45165
I20260927 22:41:27.083557 9988 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:27.084224 9988 heartbeater.cc:511] Master 127.1.53.126:45165 requested a full tablet report, sending...
I20260927 22:41:27.085649 9531 ts_manager.cc:194] Registered new tserver with Master: 361e980dd4dc478eb5a2ccaf310d9fdc (127.1.53.67:42681)
I20260927 22:41:27.086608 9531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:48497
I20260927 22:41:27.092871 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab.
Entry for principal oryx/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab.
WARNING: no policy specified for oryx/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab.
Entry for principal oryx/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab.
WARNING: no policy specified for oryx/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab.
Entry for principal oryx/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab.
I20260927 22:41:27.239367 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 9593
Sep 27 22:41:27 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:41:27.257134 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:27.240619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.1.53.65:39517)
0927 22:41:27.241590 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:27.241600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:27.241625 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:41:27.241681 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:27.241687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:27.241893 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:27.242076 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:27.243391 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.244485 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.245727 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.246314 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.246463 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:27.250675 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:27.250686 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:27.250701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:27.250733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:27.254120 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.254866 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.254874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.254878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.255024 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.255428 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.255433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.255439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.255790 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:27.256124 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:27.256474 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 22:41:27.256749 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":653,"threads_started":1}
I20260927 22:41:27.258585 9716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:27.344569 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 9593
I20260927 22:41:27.361228 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 9727
Sep 27 22:41:27 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:41:27.377683 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:27.362198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.1.53.66:33177)
0927 22:41:27.362723 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:27.362731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:27.362757 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:41:27.362822 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:27.362828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:27.363038 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:27.363170 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:27.364565 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.365604 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.366881 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.367437 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.367544 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:27.371792 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:27.371803 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:27.371818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:27.371854 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:27.375015 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.375606 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.375613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.375615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.375729 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.376074 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.376079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.376081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.376524 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:27.376767 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:27.377100 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 22:41:27.377298 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":296,"threads_started":1}
I20260927 22:41:27.378962 9850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:27.466491 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 9727
I20260927 22:41:27.482748 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 9860
Sep 27 22:41:27 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:41:27.499228 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:27.483870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.1.53.67:42681)
0927 22:41:27.484551 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:27.484558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:27.484585 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:41:27.484643 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:27.484650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:27.484883 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:27.485061 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:27.486360 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.487372 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.488678 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:27.489240 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:27.489385 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:27.493744 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:27.493754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:27.493769 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:27.493801 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:27.496507 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.497167 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.497180 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.497183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.497303 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:27.497673 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:27.497680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:27.497683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:27.498011 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:27.498327 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:27.498650 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 22:41:27.498901 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":374,"threads_started":1}
I20260927 22:41:27.500780 9983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:27.582841 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 9860
I20260927 22:41:27.600698 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 9500
I20260927 22:41:27.601816 9562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:27.704528 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 9500
I20260927 22:41:27.725971 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45165
--webserver_interface=127.1.53.126
--webserver_port=46357
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:45165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:28.045135 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:28.045426 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:28.045585 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:28.052860 10005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:28.053028 10005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:28.053138 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:28.053232 10005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:28.053315 10005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:28.063190 10005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:45165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=46357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:28.064699 10005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:28.066107 10005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:28.078044 10010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:28.078052 10011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:28.077970 10013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:28.079077 10005 server_base.cc:1034] running on GCE node
I20260927 22:41:28.079828 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:28.081290 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:28.082520 10005 hybrid_clock.cc:648] HybridClock initialized: now 1790548888082468 us; error 42 us; skew 500 ppm
I20260927 22:41:28.082868 10005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548888, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:28.086012 10005 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:41:28.087423 10005 webserver.cc:460] Webserver started at http://127.1.53.126:46357/ using document root <none> and password file <none>
I20260927 22:41:28.088104 10005 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:28.088303 10005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:28.093756 10005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:41:28.096809 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:28.098335 10005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:41:28.098523 10005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "8ae6ff051854418e9edc8209ae27f5bd"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "aec8b946f87554e7da8ff3fb0f840a99"
server_key_iv: "4bad4a1c2c4ba11f40389635b6196119"
server_key_version: "encryptionkey@0"
I20260927 22:41:28.099535 10005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:28.117038 10005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:28.121341 10005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:28.121616 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:28.141180 10005 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:45165
I20260927 22:41:28.141214 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:45165 every 8 connection(s)
I20260927 22:41:28.142736 10005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:28.146404 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10005
I20260927 22:41:28.147185 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:39517
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:45165
--webserver_port=42225
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:41:28.149873 10072 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:41:28.153628 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Bootstrap starting.
I20260927 22:41:28.173025 10072 log.cc:824] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:28.182559 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:41:28.183115 10072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Bootstrap complete.
I20260927 22:41:28.192150 10072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:28.193138 10072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae6ff051854418e9edc8209ae27f5bd, State: Initialized, Role: FOLLOWER
I20260927 22:41:28.193835 10072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:28.194089 10072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:28.194222 10072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:28.194415 10072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:41:28.198257 10072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:28.198776 10072 leader_election.cc:302] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ae6ff051854418e9edc8209ae27f5bd; no voters:
I20260927 22:41:28.199435 10072 leader_election.cc:288] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:41:28.199564 10076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:41:28.200970 10076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [term 2 LEADER]: Becoming Leader. State: Replica: 8ae6ff051854418e9edc8209ae27f5bd, State: Running, Role: LEADER
I20260927 22:41:28.201530 10076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } }
I20260927 22:41:28.201812 10072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:28.204134 10077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8ae6ff051854418e9edc8209ae27f5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } } }
I20260927 22:41:28.204499 10077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:28.204844 10078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ae6ff051854418e9edc8209ae27f5bd. Latest consensus state: current_term: 2 leader_uuid: "8ae6ff051854418e9edc8209ae27f5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ae6ff051854418e9edc8209ae27f5bd" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45165 } } }
I20260927 22:41:28.205220 10078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:28.215512 10083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:28.225317 10083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:28.226853 10083 catalog_manager.cc:1259] Loaded cluster ID: 05521c2376d74e00bf8be073d80aea4b
I20260927 22:41:28.227020 10083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:28.231076 10083 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:28.232620 10083 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8ae6ff051854418e9edc8209ae27f5bd: Loaded TSK: 0
I20260927 22:41:28.234207 10083 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:41:28.499809 10074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:28.500113 10074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:28.500298 10074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:28.508010 10074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:28.508119 10074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:28.508338 10074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:28.518347 10074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.65.keytab
--principal=oryx/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:39517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=42225
--tserver_master_addrs=127.1.53.126:45165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:28.519698 10074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:28.521116 10074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:28.536214 10099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:28.536266 10098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:28.537263 10074 server_base.cc:1034] running on GCE node
W20260927 22:41:28.537351 10101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:28.538034 10074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:28.539013 10074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:28.540325 10074 hybrid_clock.cc:648] HybridClock initialized: now 1790548888540202 us; error 53 us; skew 500 ppm
I20260927 22:41:28.540665 10074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548888, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:28.549875 10074 init.cc:377] Logged in from keytab as oryx/127.1.53.65@KRBTEST.COM (short username oryx)
I20260927 22:41:28.551779 10074 webserver.cc:460] Webserver started at http://127.1.53.65:42225/ using document root <none> and password file <none>
I20260927 22:41:28.552588 10074 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:28.552748 10074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:28.558190 10074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:41:28.561100 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:28.562595 10074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:28.562760 10074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "2cede6de25cd4387b64659406c7fa8b5"
format_stamp: "Formatted at 2026-09-27 22:41:25 on dist-test-slave-pp80"
server_key: "48148e090dd199781fb9c780f67dc045"
server_key_iv: "918d3076ceba736e4a41ebded81011c2"
server_key_version: "encryptionkey@0"
I20260927 22:41:28.563705 10074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:28.595544 10074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:28.599917 10074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:28.600200 10074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:28.601189 10074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:28.602672 10074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:28.602757 10074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:28.602856 10074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:28.602942 10074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:28.637084 10074 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:39517
I20260927 22:41:28.637226 10220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:39517 every 8 connection(s)
I20260927 22:41:28.638998 10074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:28.641902 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10074
I20260927 22:41:28.642598 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:33177
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:45165
--webserver_port=34133
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:28.661445 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:28.642403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:32819)
0927 22:41:28.644843 (+ 2440us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:28.644890 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:28.646596 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:28.648289 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:28.648302 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:28.649101 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:28.649993 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:28.650023 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.652007 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.652012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:28.653536 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:28.653550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.654688 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.654695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:28.654988 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:28.655759 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:28.655784 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:28.659976 (+ 4192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2161,"thread_start_us":1956,"threads_started":1}
W20260927 22:41:28.661463 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:28.642389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:32819 (local address 127.1.53.126:45165)
0927 22:41:28.643358 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:28.643378 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:28.645045 (+ 1667us) server_negotiation.cc:407] Connection header received
0927 22:41:28.647208 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:28.647229 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:28.647694 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:28.648102 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:28.650252 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.651807 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.653712 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.654430 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.654648 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:28.660393 (+ 5745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:32819: BlockingRecv error: recv got EOF from 127.1.53.65:32819 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":475,"threads_started":1}
W20260927 22:41:28.662195 10221 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:45165 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.53.126:45165: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:28.671380 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:28.662826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:49845)
0927 22:41:28.663014 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:28.663042 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:28.663176 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:28.663503 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:28.663508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:28.663863 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:28.664012 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:28.664021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.665184 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.665190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:28.666174 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:28.666185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.666932 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.666938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:28.667139 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:28.667811 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:28.667831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:28.671089 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:41:28.671880 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:28.663004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:49845 (local address 127.1.53.126:45165)
0927 22:41:28.663190 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:28.663196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:28.663217 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:41:28.663280 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:28.663286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:28.663362 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:28.663479 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:28.664159 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.665038 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.666336 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.666770 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.666912 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:28.671278 (+ 4366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:49845: BlockingRecv error: recv got EOF from 127.1.53.65:49845 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:28 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:28.681015 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:28.672070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:36901)
0927 22:41:28.672319 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:28.672337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:28.672450 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:28.672791 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:28.672795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:28.673155 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:28.673335 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:28.673345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.674360 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.674365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:28.675770 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:28.675784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.676561 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.676567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:28.676732 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:28.677369 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:28.677390 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:28.680721 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260927 22:41:28.681386 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:28.672388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:36901 (local address 127.1.53.126:45165)
0927 22:41:28.672532 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:28.672536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:28.672551 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:41:28.672592 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:28.672595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:28.672657 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:28.672740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:28.673468 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.674220 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.675929 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:28.676420 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:28.676524 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:28.680963 (+ 4439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:36901: BlockingRecv error: recv got EOF from 127.1.53.65:36901 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:41:28.681411 10221 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:41:28.975864 10225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:28.976176 10225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:28.976364 10225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:28.983518 10225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:28.983676 10225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:28.983867 10225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:28.993765 10225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.66.keytab
--principal=oryx/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:33177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=34133
--tserver_master_addrs=127.1.53.126:45165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:28.995128 10225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:28.996511 10225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:29.010695 10230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:29.010649 10231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:29.010720 10233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:29.011410 10225 server_base.cc:1034] running on GCE node
I20260927 22:41:29.011982 10225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:29.012976 10225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:29.014200 10225 hybrid_clock.cc:648] HybridClock initialized: now 1790548889014142 us; error 42 us; skew 500 ppm
I20260927 22:41:29.014497 10225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548889, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:29.018134 10225 init.cc:377] Logged in from keytab as oryx/127.1.53.66@KRBTEST.COM (short username oryx)
I20260927 22:41:29.019544 10225 webserver.cc:460] Webserver started at http://127.1.53.66:34133/ using document root <none> and password file <none>
I20260927 22:41:29.020213 10225 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:29.020406 10225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:29.026125 10225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:41:29.029018 10240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:29.030422 10225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:29.030555 10225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "92fe692a041f4e00aa5ccfbd7da6daad"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "640ddf767ce3b9c22976c3977299d3b1"
server_key_iv: "e92146776e25c9e206cdfea3876fec7a"
server_key_version: "encryptionkey@0"
I20260927 22:41:29.031500 10225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:29.048822 10225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:29.053115 10225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:29.053435 10225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:29.054409 10225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:29.055747 10225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:29.055848 10225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:29.055953 10225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:29.056028 10225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:29.089614 10225 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:33177
I20260927 22:41:29.089799 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:33177 every 8 connection(s)
I20260927 22:41:29.091136 10225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:29.096143 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10225
I20260927 22:41:29.096766 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:42681
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:45165
--webserver_port=45187
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:41:29.110039 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.093707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:44265 (local address 127.1.53.126:45165)
0927 22:41:29.093978 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.093985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.095232 (+ 1247us) server_negotiation.cc:407] Connection header received
0927 22:41:29.097225 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.097234 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.097337 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.097460 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.098979 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.100096 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.102118 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.102900 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.103052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.109803 (+ 6751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:44265: BlockingRecv error: recv got EOF from 127.1.53.66:44265 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:41:29.111078 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.094301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:44265)
0927 22:41:29.095058 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.095102 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.096874 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.097693 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.097701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.098225 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.098787 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.098805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.100538 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.100547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.101984 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.101995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.103185 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.103194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.103472 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.104364 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.104402 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.109602 (+ 5200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":284,"threads_started":1}
W20260927 22:41:29.111930 10353 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:45165 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.53.126:45165: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:29.120859 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.112658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:40853)
0927 22:41:29.112991 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.113006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.113150 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.113483 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.113488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.113826 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.113964 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.113972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.114966 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.114972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.115842 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.115852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.116662 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.116667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.116881 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.117684 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.117702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.120547 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:41:29.121309 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.112747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:40853 (local address 127.1.53.126:45165)
0927 22:41:29.112908 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.112934 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.113081 (+ 147us) server_negotiation.cc:407] Connection header received
0927 22:41:29.113280 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.113285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.113354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.113461 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.114108 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.114811 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.116005 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.116519 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.116650 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.120761 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:40853: BlockingRecv error: recv got EOF from 127.1.53.66:40853 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:29.130131 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.121636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:44405)
0927 22:41:29.121830 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.121851 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.121937 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.122325 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.122328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.122597 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.122733 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.122740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.124357 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.124361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.125273 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.125283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.126248 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.126252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.126407 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.127054 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.127071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.129879 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 22:41:29.130481 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.121767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:44405 (local address 127.1.53.126:45165)
0927 22:41:29.121990 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.121999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.122024 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:41:29.122087 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.122095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.122189 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.122315 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.122875 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.124176 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.125408 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.126110 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.126240 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.130179 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:44405: BlockingRecv error: recv got EOF from 127.1.53.66:44405 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:41:29.130508 10353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:41:29.430140 10356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:29.430423 10356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:29.430516 10356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:29.437922 10356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:29.438032 10356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:29.438143 10356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:29.447736 10356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/krb5kdc/oryx_127.1.53.67.keytab
--principal=oryx/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:42681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=45187
--tserver_master_addrs=127.1.53.126:45165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:29.449045 10356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:29.450248 10356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:29.463325 10361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:29.463297 10364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:29.463258 10362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:29.464541 10356 server_base.cc:1034] running on GCE node
I20260927 22:41:29.465152 10356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:29.466110 10356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:29.467335 10356 hybrid_clock.cc:648] HybridClock initialized: now 1790548889467261 us; error 54 us; skew 500 ppm
I20260927 22:41:29.467654 10356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548889, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:29.471097 10356 init.cc:377] Logged in from keytab as oryx/127.1.53.67@KRBTEST.COM (short username oryx)
I20260927 22:41:29.472767 10356 webserver.cc:460] Webserver started at http://127.1.53.67:45187/ using document root <none> and password file <none>
I20260927 22:41:29.473532 10356 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:29.473696 10356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:29.479559 10356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:41:29.482529 10371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:29.483861 10356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:41:29.484056 10356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "361e980dd4dc478eb5a2ccaf310d9fdc"
format_stamp: "Formatted at 2026-09-27 22:41:26 on dist-test-slave-pp80"
server_key: "6940b84aa04c1c79b748c973eaa6f586"
server_key_iv: "be49f03430b4f45d18a5daf6b8341951"
server_key_version: "encryptionkey@0"
I20260927 22:41:29.485090 10356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:29.503330 10356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:29.507632 10356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:29.507916 10356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:29.508996 10356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:29.510351 10356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:29.510500 10356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:29.510640 10356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:29.510727 10356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:29.544009 10356 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:42681
I20260927 22:41:29.544064 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:42681 every 8 connection(s)
I20260927 22:41:29.545557 10356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:29.553551 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10356
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:41:29.563267 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.548266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:49515 (local address 127.1.53.126:45165)
0927 22:41:29.548461 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.548467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.549652 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 22:41:29.551532 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.551536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.551604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.551694 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.553738 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.554929 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.556855 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.557359 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.557532 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.563044 (+ 5512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:49515: BlockingRecv error: recv got EOF from 127.1.53.67:49515 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:41:29.564304 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.548764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:49515)
0927 22:41:29.549472 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.549516 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.551245 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.551982 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.551991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.552772 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.553539 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.553559 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.555187 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.555193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.556690 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.556704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.557620 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.557629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.557912 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.558685 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.558710 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.562840 (+ 4130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":289,"threads_started":1}
W20260927 22:41:29.565112 10484 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:45165 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.53.126:45165: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
I20260927 22:41:29.571440 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.554555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.1.53.126:45165)
0927 22:41:29.555097 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.555105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.555441 (+ 336us) server_negotiation.cc:407] Connection header received
0927 22:41:29.555644 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.555648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.555703 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.555810 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.557341 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.558160 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.559421 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.559889 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.560025 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.562965 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:29.562977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:29.563002 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:29.563052 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:29.567650 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:29.568557 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:29.568565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:29.568572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:29.568777 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:29.569240 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:29.569244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:29.569246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:29.569701 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:29.570160 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:29.570571 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 22:41:29.570979 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":306,"threads_started":1}
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:29.576082 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.565910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:47379)
0927 22:41:29.566116 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.566134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.566238 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.566708 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.566712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.567019 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.567171 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.567179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.568599 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.568603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.569684 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.569695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.570890 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.570896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.571092 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.572439 (+ 1347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.572464 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.575852 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 22:41:29.576344 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.566016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:47379 (local address 127.1.53.126:45165)
0927 22:41:29.566196 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.566202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.566220 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:41:29.566407 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.566414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.566519 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.566638 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.567345 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.568366 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.569822 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.570584 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.570705 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.576052 (+ 5347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:47379: BlockingRecv error: recv got EOF from 127.1.53.67:47379 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:29.584545 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.576841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:35193)
0927 22:41:29.577009 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.577031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.577108 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.577447 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.577450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.577716 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.577865 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.577873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.578918 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.578922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.579848 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.579859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.580642 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.580646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.580794 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.581365 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.581382 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.584237 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:41:29.584858 10484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:41:29.585091 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.576983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:35193 (local address 127.1.53.126:45165)
0927 22:41:29.577208 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.577212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.577227 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:41:29.577268 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.577271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.577327 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.577411 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.577984 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.578768 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.579997 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.580510 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.580610 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.584499 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:35193: BlockingRecv error: recv got EOF from 127.1.53.67:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:29 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:29.690969 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:29.682130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:33503)
0927 22:41:29.682751 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:29.682767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:29.682887 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:29.683260 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:29.683267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:29.683888 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:29.684148 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.684162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.685226 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.685230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.686184 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:29.686195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.686898 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.686902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:29.687022 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.687609 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:29.687626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:29.690747 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":372,"threads_started":1}
W20260927 22:41:29.691262 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:29.682255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:33503 (local address 127.1.53.126:45165)
0927 22:41:29.682466 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:29.682473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:29.682852 (+ 379us) server_negotiation.cc:407] Connection header received
0927 22:41:29.683017 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:29.683024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:29.683105 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:29.683192 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:29.684328 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.685095 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.686320 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:29.686751 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:29.686846 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:29.690969 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:33503: BlockingRecv error: recv got EOF from 127.1.53.65:33503 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:41:30 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:30 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:30.147564 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:30.131258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:33469)
0927 22:41:30.131990 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:30.132031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:30.132211 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:30.132630 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:30.132634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:30.133176 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:30.133403 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:30.133416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.134779 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.134783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:30.135961 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:30.135974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.136785 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.136790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:30.136941 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:30.143485 (+ 6544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:30.143504 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:30.147221 (+ 3717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":463,"threads_started":1}
W20260927 22:41:30.147600 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:30.131325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:33469 (local address 127.1.53.126:45165)
0927 22:41:30.131654 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:30.131662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:30.132154 (+ 492us) server_negotiation.cc:407] Connection header received
0927 22:41:30.132395 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:30.132420 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:30.132496 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:30.132598 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:30.133556 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.134619 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.136112 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.136605 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.136712 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:30.147395 (+ 10683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:33469: BlockingRecv error: recv got EOF from 127.1.53.66:33469 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 27 22:41:30 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:30 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:30.593999 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:30.585526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:54895)
0927 22:41:30.586076 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:30.586092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:30.586205 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:30.586498 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:30.586502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:30.586932 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:30.587146 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:30.587154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.588313 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.588318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:30.589539 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:30.589552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.590298 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.590305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:30.590477 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:30.591043 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:30.591060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:30.593768 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":292,"threads_started":1}
W20260927 22:41:30.594218 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:30.585571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:54895 (local address 127.1.53.126:45165)
0927 22:41:30.585762 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:30.585766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:30.586176 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:41:30.586304 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:30.586308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:30.586371 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:30.586455 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:30.587298 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.588088 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.589682 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.590119 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.590258 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:30.593966 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:54895: BlockingRecv error: recv got EOF from 127.1.53.67:54895 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 22:41:30 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:30 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:30.700404 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:30.691889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:55303)
0927 22:41:30.692488 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:30.692504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:30.692671 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:30.693080 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:30.693086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:30.693501 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:30.693637 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:30.693645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.694669 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.694674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:30.695582 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:30.695593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.696338 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.696344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:30.696511 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:30.697171 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:30.697189 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:30.700096 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":365,"threads_started":1}
W20260927 22:41:30.700757 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:30.692033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:55303 (local address 127.1.53.126:45165)
0927 22:41:30.692341 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:30.692348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:30.692593 (+ 245us) server_negotiation.cc:407] Connection header received
0927 22:41:30.692809 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:30.692869 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:30.692934 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:30.693046 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:30.693779 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.694527 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.695734 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:30.696143 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:30.696306 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:30.700365 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:55303: BlockingRecv error: recv got EOF from 127.1.53.65:55303 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 22:41:31 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:31 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:31.156766 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:31.148491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:57083)
0927 22:41:31.148966 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:31.148982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:31.149116 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:31.149460 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:31.149465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:31.149916 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:31.150059 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:31.150068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.151087 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.151094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:31.152016 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:31.152026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.152790 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.152796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:31.152974 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:31.153602 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:31.153619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:31.156527 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
W20260927 22:41:31.157094 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:31.148663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:57083 (local address 127.1.53.126:45165)
0927 22:41:31.148879 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:31.148885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:31.149067 (+ 182us) server_negotiation.cc:407] Connection header received
0927 22:41:31.149232 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:31.149235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:31.149312 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:31.149393 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:31.150202 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.150950 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.152153 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.152641 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.152777 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:31.156763 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:57083: BlockingRecv error: recv got EOF from 127.1.53.66:57083 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 22:41:31 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:31 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:31.603600 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:31.594933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:56721)
0927 22:41:31.595502 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:31.595518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:31.595648 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:31.595978 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:31.595981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:31.596372 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:31.596513 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:31.596521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.597687 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.597692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:31.598633 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:31.598645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.599415 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.599420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:31.599542 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:31.600233 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:31.600267 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:31.603309 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":347,"threads_started":1}
W20260927 22:41:31.603840 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:31.595138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:56721 (local address 127.1.53.126:45165)
0927 22:41:31.595320 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:31.595324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:31.595638 (+ 314us) server_negotiation.cc:407] Connection header received
0927 22:41:31.595745 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:31.595763 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:31.595832 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:31.595935 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:31.596666 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.597539 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.598772 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.599271 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.599375 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:31.603543 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:56721: BlockingRecv error: recv got EOF from 127.1.53.67:56721 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 22:41:31 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:31 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:31.710289 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:31.701329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:42827)
0927 22:41:31.701888 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:31.701948 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:31.702086 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:31.702409 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:31.702413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:31.702741 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:31.702916 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:31.702925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.704222 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.704229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:31.705294 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:31.705307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.706103 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.706108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:31.706243 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:31.706953 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:31.706977 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:31.709893 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":294,"threads_started":1}
W20260927 22:41:31.710384 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:31.701416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:42827 (local address 127.1.53.126:45165)
0927 22:41:31.701602 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:31.701606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:31.702034 (+ 428us) server_negotiation.cc:407] Connection header received
0927 22:41:31.702233 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:31.702237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:31.702300 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:31.702388 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:31.703122 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.704019 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.705446 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:31.705864 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:31.706026 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:31.710115 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:42827: BlockingRecv error: recv got EOF from 127.1.53.65:42827 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:41:32 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:32 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:32.166235 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:32.158064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:34405)
0927 22:41:32.158606 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:32.158626 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:32.158789 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:32.159149 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:32.159152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:32.159477 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:32.159651 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:32.159659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.160792 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.160796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:32.161700 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:32.161712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.162487 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.162492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:32.162617 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:32.163281 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:32.163298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:32.165970 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":250,"threads_started":1}
W20260927 22:41:32.166380 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:32.158119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:34405 (local address 127.1.53.126:45165)
0927 22:41:32.158315 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:32.158323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:32.158722 (+ 399us) server_negotiation.cc:407] Connection header received
0927 22:41:32.158943 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:32.158949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:32.159026 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:32.159103 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:32.159804 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.160653 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.161855 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.162342 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.162487 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:32.166154 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:34405: BlockingRecv error: recv got EOF from 127.1.53.66:34405 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:41:32 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:32 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:32.614997 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:32.604648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:60817)
0927 22:41:32.606678 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:32.606694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:32.606843 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:32.607395 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:32.607414 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:32.607851 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:32.608028 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:32.608036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.609331 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.609338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:32.610254 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:32.610266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.610942 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.610948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:32.611136 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:32.611776 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:32.611794 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:32.614628 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1910,"thread_start_us":1760,"threads_started":1}
W20260927 22:41:32.615056 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:32.604714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:60817 (local address 127.1.53.126:45165)
0927 22:41:32.604914 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:32.604922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:32.606776 (+ 1854us) server_negotiation.cc:407] Connection header received
0927 22:41:32.606989 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:32.606997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:32.607085 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:32.607176 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:32.608171 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.609202 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.610394 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.610800 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.610939 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:32.614822 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:60817: BlockingRecv error: recv got EOF from 127.1.53.67:60817 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 22:41:32 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:32 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:32.720336 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:32.711332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:46033)
0927 22:41:32.711808 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:32.711825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:32.711970 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:32.712291 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:32.712294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:32.712671 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:32.712909 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:32.712956 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.714082 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.714086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:32.715048 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:32.715059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.715872 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.715879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:32.716100 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:32.716906 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:32.716952 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:32.719978 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":274,"threads_started":1}
W20260927 22:41:32.720398 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:32.711406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:46033 (local address 127.1.53.126:45165)
0927 22:41:32.711682 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:32.711688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:32.711947 (+ 259us) server_negotiation.cc:407] Connection header received
0927 22:41:32.712076 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:32.712080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:32.712143 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:32.712224 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:32.713092 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.713912 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.715200 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:32.715699 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:32.715818 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:32.720136 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:46033: BlockingRecv error: recv got EOF from 127.1.53.65:46033 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 22:41:33 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:33 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:33.175747 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:33.167248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:42461)
0927 22:41:33.167761 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:33.167776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:33.167930 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:33.168310 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:33.168314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:33.168711 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:33.168851 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:33.168859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.169832 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.169839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:33.170810 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:33.170821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.171524 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.171530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:33.171719 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:33.172312 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:33.172329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:33.175508 (+ 3179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":295,"threads_started":1}
W20260927 22:41:33.175894 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:33.167288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:42461 (local address 127.1.53.126:45165)
0927 22:41:33.167468 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:33.167472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:33.167867 (+ 395us) server_negotiation.cc:407] Connection header received
0927 22:41:33.168080 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:33.168090 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:33.168149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:33.168278 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:33.169001 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.169699 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.170951 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.171368 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.171503 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:33.175698 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:42461: BlockingRecv error: recv got EOF from 127.1.53.66:42461 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:41:33 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:33 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:33.630664 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:33.615993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:51349)
0927 22:41:33.616553 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:33.616571 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:33.616743 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:33.617164 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:33.617169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:33.617699 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:33.617900 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:33.617914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.619032 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.619036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:33.620021 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:33.620032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.620752 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.620756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:33.620922 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:33.627310 (+ 6388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:33.627330 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:33.630434 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
W20260927 22:41:33.630867 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:33.616062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:51349 (local address 127.1.53.126:45165)
0927 22:41:33.616324 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:33.616333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:33.616687 (+ 354us) server_negotiation.cc:407] Connection header received
0927 22:41:33.616913 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:33.616918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:33.617004 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:33.617103 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:33.618037 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.618880 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.620151 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.620601 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.620697 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:33.630621 (+ 9924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:51349: BlockingRecv error: recv got EOF from 127.1.53.67:51349 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:41:33 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:33 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:33.730044 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:33.721401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:51805)
0927 22:41:33.721931 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:33.721948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:33.722059 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:33.722362 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:33.722365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:33.722732 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:33.722887 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:33.722895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.724018 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.724025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:33.725467 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:33.725490 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.726194 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.726197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:33.726313 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:33.726970 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:33.726989 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:33.729788 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":274,"threads_started":1}
W20260927 22:41:33.730327 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:33.721524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:51805 (local address 127.1.53.126:45165)
0927 22:41:33.721691 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:33.721696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:33.722036 (+ 340us) server_negotiation.cc:407] Connection header received
0927 22:41:33.722161 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:33.722165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:33.722226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:33.722316 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:33.723015 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.723878 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.725627 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:33.726070 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:33.726172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:33.730058 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:51805: BlockingRecv error: recv got EOF from 127.1.53.65:51805 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:41:34 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:34 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:34.185243 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:34.176892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:49397)
0927 22:41:34.177475 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:34.177495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:34.177622 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:34.178039 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:34.178044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:34.178423 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:34.178571 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:34.178578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.179583 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.179587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:34.180547 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:34.180557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.181311 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.181315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:34.181430 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:34.182070 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:34.182086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:34.184966 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":255,"threads_started":1}
W20260927 22:41:34.185549 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:34.177111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:49397 (local address 127.1.53.126:45165)
0927 22:41:34.177307 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:34.177313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:34.177565 (+ 252us) server_negotiation.cc:407] Connection header received
0927 22:41:34.177758 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:34.177761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:34.177856 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:34.178006 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:34.178717 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.179469 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.180695 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.181179 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.181290 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:34.185223 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:49397: BlockingRecv error: recv got EOF from 127.1.53.66:49397 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:41:34 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:34 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:34.641800 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:34.631664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:55141)
0927 22:41:34.632204 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:34.632219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:34.632391 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:34.632963 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:34.632968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:34.633402 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:34.633619 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:34.633638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.635088 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.635093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:34.636303 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:34.636317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.637221 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.637227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:34.637440 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:34.638144 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:34.638165 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:34.641555 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":276,"threads_started":1}
W20260927 22:41:34.642006 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:34.631751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:55141 (local address 127.1.53.126:45165)
0927 22:41:34.632007 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:34.632012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:34.632323 (+ 311us) server_negotiation.cc:407] Connection header received
0927 22:41:34.632619 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:34.632625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:34.632741 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:34.632868 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:34.633772 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.634940 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.636454 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.637010 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.637123 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:34.641744 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:55141: BlockingRecv error: recv got EOF from 127.1.53.67:55141 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 22:41:34 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:34 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:34.739441 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:34.731078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:36607)
0927 22:41:34.731606 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:34.731626 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:34.731784 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:34.732184 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:34.732190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:34.732609 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:34.732768 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:34.732777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.733795 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.733800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:34.734746 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:34.734758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.735462 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.735468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:34.735656 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:34.736266 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:34.736285 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:34.739213 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
W20260927 22:41:34.739728 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:34.731164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:36607 (local address 127.1.53.126:45165)
0927 22:41:34.731408 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:34.731416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:34.731717 (+ 301us) server_negotiation.cc:407] Connection header received
0927 22:41:34.731926 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:34.731973 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:34.732040 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:34.732146 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:34.732908 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.733654 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.734883 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:34.735302 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:34.735437 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:34.739388 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:36607: BlockingRecv error: recv got EOF from 127.1.53.65:36607 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 22:41:35 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:35 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:35.195307 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:35.186216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:38109)
0927 22:41:35.186849 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:35.186864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:35.187028 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:35.187398 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:35.187403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:35.187815 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:35.187971 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:35.187979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.189375 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.189381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:35.190728 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:35.190741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.191401 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.191405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:35.191517 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:35.192190 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:35.192207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:35.195088 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":388,"threads_started":1}
W20260927 22:41:35.195621 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:35.186378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:38109 (local address 127.1.53.126:45165)
0927 22:41:35.186627 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:35.186635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:35.186962 (+ 327us) server_negotiation.cc:407] Connection header received
0927 22:41:35.187182 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:35.187187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:35.187262 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:35.187370 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:35.188142 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.189214 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.190860 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.191270 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.191368 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:35.195320 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:38109: BlockingRecv error: recv got EOF from 127.1.53.66:38109 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:41:35 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:35 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:35.651194 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:35.642911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:57293)
0927 22:41:35.643453 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:35.643469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:35.643583 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:35.643864 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:35.643867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:35.644229 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:35.644434 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:35.644443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.645483 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.645488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:35.646403 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:35.646414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.647185 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.647190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:35.647328 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:35.647994 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:35.648011 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:35.650876 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":314,"threads_started":1}
W20260927 22:41:35.651307 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:35.643011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:57293 (local address 127.1.53.126:45165)
0927 22:41:35.643179 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:35.643184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:35.643543 (+ 359us) server_negotiation.cc:407] Connection header received
0927 22:41:35.643678 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:35.643682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:35.643741 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:35.643820 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:35.644572 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.645303 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.646555 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.647021 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.647124 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:35.651069 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:57293: BlockingRecv error: recv got EOF from 127.1.53.67:57293 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:41:35 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:35 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:35.748777 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:35.740383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:41793)
0927 22:41:35.740924 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:35.740942 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:35.741073 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:35.741362 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:35.741365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:35.741811 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:35.741980 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:35.741991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.743213 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.743217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:35.744202 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:35.744215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.745017 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.745022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:35.745146 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:35.745753 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:35.745771 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:35.748544 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":264,"threads_started":1}
W20260927 22:41:35.749001 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:35.740470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:41793 (local address 127.1.53.126:45165)
0927 22:41:35.740709 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:35.740719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:35.741091 (+ 372us) server_negotiation.cc:407] Connection header received
0927 22:41:35.741164 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:35.741168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:35.741229 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:35.741312 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:35.742127 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.743070 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.744419 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:35.744832 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:35.744931 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:35.748754 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:41793: BlockingRecv error: recv got EOF from 127.1.53.65:41793 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 22:41:36 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:36 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:36.205390 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:36.196447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:38937)
0927 22:41:36.197015 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:36.197032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:36.197148 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:36.197524 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:36.197527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:36.197982 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:36.198145 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:36.198153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.199155 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.199159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:36.200071 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:36.200082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.200921 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.200928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:36.201086 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:36.201805 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:36.201823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:36.205140 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":261,"threads_started":1}
W20260927 22:41:36.205607 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:36.196465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:38937 (local address 127.1.53.126:45165)
0927 22:41:36.196651 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:36.196655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:36.197110 (+ 455us) server_negotiation.cc:407] Connection header received
0927 22:41:36.197297 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:36.197302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:36.197387 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:36.197478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:36.198288 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.199034 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.200209 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.200725 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.200821 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:36.205334 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:38937: BlockingRecv error: recv got EOF from 127.1.53.66:38937 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:41:36 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:36 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:36.660631 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:36.652311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:33855)
0927 22:41:36.652876 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:36.652901 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:36.653012 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:36.653363 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:36.653367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:36.653810 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:36.653987 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:36.653996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.655017 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.655021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:36.655924 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:36.655934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.656718 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.656722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:36.656835 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:36.657471 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:36.657488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:36.660384 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":261,"threads_started":1}
W20260927 22:41:36.660868 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:36.652329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:33855 (local address 127.1.53.126:45165)
0927 22:41:36.652523 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:36.652528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:36.652991 (+ 463us) server_negotiation.cc:407] Connection header received
0927 22:41:36.653115 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:36.653119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:36.653181 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:36.653272 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:36.654119 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.654898 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.656063 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.656584 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.656683 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:36.660599 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:33855: BlockingRecv error: recv got EOF from 127.1.53.67:33855 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:41:36 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:36 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:36.757776 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:36.749677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:36059)
0927 22:41:36.750169 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:36.750184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:36.750294 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:36.750583 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:36.750586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:36.750951 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:36.751111 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:36.751119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.752135 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.752139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:36.753126 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:36.753138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.753859 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.753863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:36.753973 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:36.754608 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:36.754628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:36.757523 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
W20260927 22:41:36.757958 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:36.749799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:36059 (local address 127.1.53.126:45165)
0927 22:41:36.750021 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:36.750027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:36.750261 (+ 234us) server_negotiation.cc:407] Connection header received
0927 22:41:36.750398 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:36.750402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:36.750463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:36.750542 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:36.751247 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.751991 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.753254 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:36.753703 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:36.753809 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:36.757729 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:36059: BlockingRecv error: recv got EOF from 127.1.53.65:36059 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 22:41:37 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:37 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:37.215708 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:37.206430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:47747)
0927 22:41:37.206918 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:37.206935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:37.207075 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:37.207420 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:37.207424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:37.207948 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:37.208194 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:37.208205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.209665 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.209672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:37.210770 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:37.210785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.211728 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.211752 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:37.211931 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:37.212572 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:37.212588 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:37.215369 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":241,"threads_started":1}
W20260927 22:41:37.215797 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:37.206599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:47747 (local address 127.1.53.126:45165)
0927 22:41:37.206829 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:37.206836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:37.207030 (+ 194us) server_negotiation.cc:407] Connection header received
0927 22:41:37.207215 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:37.207219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:37.207285 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:37.207371 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:37.208390 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.209517 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.210927 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.211524 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.211674 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:37.215580 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:47747: BlockingRecv error: recv got EOF from 127.1.53.66:47747 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 22:41:37 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:37 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:37.670091 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:37.661555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:45417)
0927 22:41:37.662093 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:37.662110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:37.662219 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:37.662530 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:37.662533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:37.662888 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:37.663066 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:37.663075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.664048 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.664052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:37.664966 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:37.664985 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.665753 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.665757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:37.665885 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:37.666499 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:37.666517 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:37.669737 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":254,"threads_started":1}
W20260927 22:41:37.670248 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:37.661577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:45417 (local address 127.1.53.126:45165)
0927 22:41:37.661756 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:37.661760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:37.662186 (+ 426us) server_negotiation.cc:407] Connection header received
0927 22:41:37.662316 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:37.662320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:37.662397 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:37.662488 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:37.663185 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.663894 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.665106 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.665597 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.665692 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:37.669940 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:45417: BlockingRecv error: recv got EOF from 127.1.53.67:45417 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:41:37 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:37 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:41:37.767103 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:37.758883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:49657 (local address 127.1.53.126:45165)
0927 22:41:37.759143 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:37.759149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:37.759308 (+ 159us) server_negotiation.cc:407] Connection header received
0927 22:41:37.759442 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:37.759445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:37.759503 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:37.759584 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:37.760403 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.761127 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.762416 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.762835 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.762934 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:37.766910 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:49657: BlockingRecv error: recv got EOF from 127.1.53.65:49657 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:41:37.767220 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:37.758713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:49657)
0927 22:41:37.759212 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:37.759229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:37.759342 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:37.759629 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:37.759632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:37.760052 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:37.760221 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:37.760231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.761310 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.761315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:37.762258 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:37.762270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:37.762984 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:37.762988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:37.763109 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:37.763730 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:37.763747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:37.766711 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":295,"threads_started":1}
Sep 27 22:41:38 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:38 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:38.225720 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:38.216922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:48943)
0927 22:41:38.217402 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:38.217418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:38.217583 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:38.217937 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:38.217940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:38.218341 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:38.218525 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:38.218533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.219813 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.219819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:38.221015 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:38.221027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.221733 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.221739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:38.221914 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:38.222499 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:38.222516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:38.225469 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":284,"threads_started":1}
W20260927 22:41:38.225941 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:38.216980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:48943 (local address 127.1.53.126:45165)
0927 22:41:38.217220 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:38.217230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:38.217507 (+ 277us) server_negotiation.cc:407] Connection header received
0927 22:41:38.217704 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:38.217707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:38.217770 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:38.217876 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:38.218673 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.219676 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.221159 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.221577 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.221719 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:38.225668 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:48943: BlockingRecv error: recv got EOF from 127.1.53.66:48943 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 22:41:38 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:38 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:38.688083 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:38.671247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:40249)
0927 22:41:38.671736 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:38.671751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:38.671863 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:38.672159 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:38.672163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:38.672838 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:38.673077 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:38.673092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.674958 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.674963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:38.676456 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:38.676467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.677127 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.677131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:38.677258 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:38.684487 (+ 7229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:38.684506 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:38.687796 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":283,"threads_started":1}
W20260927 22:41:38.688302 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:38.671301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:40249 (local address 127.1.53.126:45165)
0927 22:41:38.671529 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:38.671539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:38.671834 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:41:38.671970 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:38.671974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:38.672036 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:38.672118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:38.673258 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.674464 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.676586 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.676989 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.677085 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:38.687991 (+ 10906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:40249: BlockingRecv error: recv got EOF from 127.1.53.67:40249 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:41:38 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:38 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:38.777817 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:38.768479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:57123)
0927 22:41:38.769435 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:38.769457 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:38.769607 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:38.770072 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:38.770077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:38.770633 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:38.770841 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:38.770854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.772073 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.772077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:38.773014 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:38.773025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.773756 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.773761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:38.773876 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:38.774503 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:38.774520 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:38.777522 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":702,"threads_started":1}
W20260927 22:41:38.777962 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:38.768604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:57123 (local address 127.1.53.126:45165)
0927 22:41:38.768859 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:38.768866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:38.769566 (+ 700us) server_negotiation.cc:407] Connection header received
0927 22:41:38.769772 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:38.769778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:38.769878 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:38.769991 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:38.771082 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.771897 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.773159 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:38.773625 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:38.773721 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:38.777698 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:57123: BlockingRecv error: recv got EOF from 127.1.53.65:57123 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:41:39 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:39 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:39.236186 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:39.226735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:34987)
0927 22:41:39.227285 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:39.227301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:39.227479 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:39.227865 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:39.227870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:39.228354 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:39.228501 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:39.228510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.229734 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.229740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:39.230918 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:39.230929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.231704 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.231711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:39.231911 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:39.232656 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:39.232675 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:39.235951 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":308,"threads_started":1}
W20260927 22:41:39.236433 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:39.226829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:34987 (local address 127.1.53.126:45165)
0927 22:41:39.227093 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:39.227102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:39.227407 (+ 305us) server_negotiation.cc:407] Connection header received
0927 22:41:39.227628 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:39.227633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:39.227708 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:39.227824 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:39.228654 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.229578 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.231065 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.231542 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.231685 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:39.236157 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:34987: BlockingRecv error: recv got EOF from 127.1.53.66:34987 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 22:41:39 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:39 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:39.697945 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:39.689052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:34571)
0927 22:41:39.689579 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:39.689605 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:39.689759 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:39.690229 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:39.690233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:39.690639 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:39.690791 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:39.690800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.691933 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.691938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:39.693122 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:39.693135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.693820 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.693826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:39.694130 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:39.694873 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:39.694892 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:39.697726 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":270,"threads_started":1}
W20260927 22:41:39.698166 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:39.689280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:34571 (local address 127.1.53.126:45165)
0927 22:41:39.689545 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:39.689551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:39.689687 (+ 136us) server_negotiation.cc:407] Connection header received
0927 22:41:39.689993 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:39.689997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:39.690061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:39.690146 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:39.690999 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.691752 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.693275 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.693665 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.693757 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:39.697913 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:34571: BlockingRecv error: recv got EOF from 127.1.53.67:34571 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 27 22:41:39 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:39 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:39.787699 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:39.778861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:34789)
0927 22:41:39.779419 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:39.779436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:39.779582 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:39.780004 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:39.780012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:39.780545 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:39.780745 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:39.780754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.781871 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.781876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:39.782918 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:39.782931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.783764 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.783769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:39.783885 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:39.784621 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:39.784641 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:39.787443 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":322,"threads_started":1}
W20260927 22:41:39.787930 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:39.778966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:34789 (local address 127.1.53.126:45165)
0927 22:41:39.779206 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:39.779213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:39.779500 (+ 287us) server_negotiation.cc:407] Connection header received
0927 22:41:39.779736 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:39.779742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:39.779826 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:39.779942 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:39.780886 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.781659 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.783077 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:39.783608 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:39.783711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:39.787657 (+ 3946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:34789: BlockingRecv error: recv got EOF from 127.1.53.65:34789 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:41:40 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:40 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:40.246246 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:40.237165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:42671)
0927 22:41:40.237703 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:40.237720 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:40.237885 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:40.238244 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:40.238248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:40.238644 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:40.238823 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:40.238831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.239864 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.239878 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:40.241043 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:40.241058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.241703 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.241707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:40.241828 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:40.242619 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:40.242640 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:40.245988 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":321,"threads_started":1}
W20260927 22:41:40.246410 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:40.237246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:42671 (local address 127.1.53.126:45165)
0927 22:41:40.237429 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:40.237433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:40.237807 (+ 374us) server_negotiation.cc:407] Connection header received
0927 22:41:40.237992 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:40.237996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:40.238076 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:40.238165 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:40.238954 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.239703 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.241183 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.241567 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.241662 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:40.246194 (+ 4532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:42671: BlockingRecv error: recv got EOF from 127.1.53.66:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:41:40 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:40 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:40.707546 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:40.698891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:33479)
0927 22:41:40.699400 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:40.699415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:40.699586 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:40.699929 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:40.699933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:40.700463 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:40.700631 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:40.700639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.701632 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.701639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:40.702616 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:40.702626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.703442 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.703448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:40.703617 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:40.704369 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:40.704390 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:40.707298 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":283,"threads_started":1}
W20260927 22:41:40.707718 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:40.698980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:33479 (local address 127.1.53.126:45165)
0927 22:41:40.699246 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:40.699262 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:40.699512 (+ 250us) server_negotiation.cc:407] Connection header received
0927 22:41:40.699736 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:40.699740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:40.699801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:40.699903 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:40.700771 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.701505 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.702769 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.703282 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.703405 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:40.707494 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:33479: BlockingRecv error: recv got EOF from 127.1.53.67:33479 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 22:41:40 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:40 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:40.797477 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:40.788838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:38941)
0927 22:41:40.789427 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:40.789442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:40.789648 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:40.789989 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:40.789999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:40.790362 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:40.790516 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:40.790525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.791466 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.791470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:40.792604 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:40.792619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.793622 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.793629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:40.793812 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:40.794581 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:40.794599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:40.797250 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":387,"threads_started":1}
W20260927 22:41:40.797669 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:40.788909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:38941 (local address 127.1.53.126:45165)
0927 22:41:40.789113 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:40.789118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:40.789574 (+ 456us) server_negotiation.cc:407] Connection header received
0927 22:41:40.789774 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:40.789778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:40.789839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:40.789922 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:40.790649 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.791340 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.792783 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:40.793456 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:40.793600 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:40.797434 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:38941: BlockingRecv error: recv got EOF from 127.1.53.65:38941 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 22:41:41 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:41 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:41.255950 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:41.247149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:50695)
0927 22:41:41.247682 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:41.247704 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:41.247813 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:41.248185 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:41.248189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:41.248606 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:41.248751 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:41.248758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.250027 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.250032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:41.251267 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:41.251280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.251994 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.251999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:41.252131 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:41.252801 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:41.252818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:41.255707 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":323,"threads_started":1}
W20260927 22:41:41.256131 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:41.247242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:50695 (local address 127.1.53.126:45165)
0927 22:41:41.247436 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:41.247441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:41.247777 (+ 336us) server_negotiation.cc:407] Connection header received
0927 22:41:41.247973 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:41.247978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:41.248043 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:41.248135 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:41.248890 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.249868 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.251415 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.251807 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.251907 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:41.255908 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:50695: BlockingRecv error: recv got EOF from 127.1.53.66:50695 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:41:41 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:41 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:41.716804 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:41.708526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:52613)
0927 22:41:41.709004 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:41.709020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:41.709199 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:41.709544 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:41.709548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:41.709985 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:41.710141 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:41.710149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.711155 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.711161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:41.712081 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:41.712091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.712855 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.712860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:41.713045 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:41.713629 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:41.713646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:41.716483 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
W20260927 22:41:41.716984 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:41.708593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:52613 (local address 127.1.53.126:45165)
0927 22:41:41.708802 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:41.708809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:41.709128 (+ 319us) server_negotiation.cc:407] Connection header received
0927 22:41:41.709323 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:41.709328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:41.709401 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:41.709505 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:41.710286 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.710999 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.712221 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.712696 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.712829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:41.716737 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:52613: BlockingRecv error: recv got EOF from 127.1.53.67:52613 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:41:41 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:41 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:41.807331 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:41.798470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:49679)
0927 22:41:41.799001 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:41.799019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:41.799137 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:41.799437 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:41.799440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:41.799823 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:41.799979 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:41.799988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.801121 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.801125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:41.802071 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:41.802082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.802900 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.802905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:41.803082 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:41.803853 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:41.803879 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:41.807113 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
W20260927 22:41:41.807520 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:41.798653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:49679 (local address 127.1.53.126:45165)
0927 22:41:41.798905 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:41.798911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:41.799110 (+ 199us) server_negotiation.cc:407] Connection header received
0927 22:41:41.799237 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:41.799241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:41.799302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:41.799391 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:41.800156 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.800959 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.802208 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:41.802696 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:41.802828 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:41.807276 (+ 4448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:49679: BlockingRecv error: recv got EOF from 127.1.53.65:49679 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 22:41:42 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:42 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:42.265962 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:42.257034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:54955)
0927 22:41:42.257561 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:42.257584 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:42.257726 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:42.258097 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:42.258102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:42.258677 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:42.258908 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:42.258920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.260472 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.260476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:42.261427 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:42.261438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.262096 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.262100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:42.262210 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:42.262760 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:42.262776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:42.265715 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":318,"threads_started":1}
W20260927 22:41:42.266160 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:42.257188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:54955 (local address 127.1.53.126:45165)
0927 22:41:42.257348 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:42.257356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:42.257694 (+ 338us) server_negotiation.cc:407] Connection header received
0927 22:41:42.257857 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:42.257862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:42.257944 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:42.258051 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:42.259064 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.260339 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.261554 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.261942 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.262039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:42.265913 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:54955: BlockingRecv error: recv got EOF from 127.1.53.66:54955 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 22:41:42 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:42 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:42.726534 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:42.717745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:50651)
0927 22:41:42.718308 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:42.718325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:42.718506 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:42.718847 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:42.718851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:42.719255 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:42.719414 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:42.719422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.720460 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.720466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:42.721486 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:42.721499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.722622 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.722629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:42.722793 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:42.723479 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:42.723497 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:42.726285 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":300,"threads_started":1}
W20260927 22:41:42.726827 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:42.717834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:50651 (local address 127.1.53.126:45165)
0927 22:41:42.718106 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:42.718113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:42.718433 (+ 320us) server_negotiation.cc:407] Connection header received
0927 22:41:42.718645 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:42.718649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:42.718710 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:42.718813 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:42.719556 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.720334 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.721644 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.722305 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.722441 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:42.726573 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:50651: BlockingRecv error: recv got EOF from 127.1.53.67:50651 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 27 22:41:42 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:42 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:42.816334 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:42.808383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:42533)
0927 22:41:42.808907 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:42.808924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:42.809039 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:42.809323 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:42.809327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:42.809676 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:42.809901 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:42.809912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.810941 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.810945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:42.811855 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:42.811866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.812590 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.812594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:42.812709 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:42.813312 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:42.813331 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:42.816060 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
W20260927 22:41:42.816570 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:42.808390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:42533 (local address 127.1.53.126:45165)
0927 22:41:42.808642 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:42.808646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:42.809006 (+ 360us) server_negotiation.cc:407] Connection header received
0927 22:41:42.809137 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:42.809140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:42.809198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:42.809277 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:42.810029 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.810773 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.811988 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:42.812469 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:42.812567 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:42.816316 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:42533: BlockingRecv error: recv got EOF from 127.1.53.65:42533 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 27 22:41:43 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:43 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:43.276624 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:43.267223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:59233)
0927 22:41:43.268158 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:43.268176 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:43.268366 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:43.268784 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:43.268788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:43.269201 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:43.269384 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:43.269394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.270517 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.270522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:43.271447 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:43.271460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.272346 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.272353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:43.272524 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:43.273184 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:43.273213 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:43.276356 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":673,"threads_started":1}
W20260927 22:41:43.276748 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:43.267324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:59233 (local address 127.1.53.126:45165)
0927 22:41:43.267600 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:43.267610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:43.268273 (+ 663us) server_negotiation.cc:407] Connection header received
0927 22:41:43.268513 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:43.268519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:43.268628 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:43.268740 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:43.269557 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.270344 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.271607 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.272107 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.272221 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:43.276524 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:59233: BlockingRecv error: recv got EOF from 127.1.53.66:59233 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:41:43 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:43 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.67@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:43.747593 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:43.727525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.67:37635)
0927 22:41:43.727991 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:43.728008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:43.728121 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:43.728444 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:43.728447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:43.728799 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:43.728973 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:43.728981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.730087 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.730091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:43.731020 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:43.731032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.731836 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.731841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:43.732014 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:43.744461 (+ 12447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:43.744483 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:43.747332 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
W20260927 22:41:43.747856 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:43.727601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:37635 (local address 127.1.53.126:45165)
0927 22:41:43.727824 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:43.727831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:43.728091 (+ 260us) server_negotiation.cc:407] Connection header received
0927 22:41:43.728216 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:43.728219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:43.728322 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:43.728405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:43.729145 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.729913 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.731183 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.731665 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.731780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:43.747578 (+ 15798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.67:37635: BlockingRecv error: recv got EOF from 127.1.53.67:37635 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:41:43 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:43 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.65@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:43.825835 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:43.817537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.65:57941)
0927 22:41:43.818053 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:43.818073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:43.818215 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:43.818568 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:43.818573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:43.818996 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:43.819178 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:43.819187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.820197 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.820202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:43.821183 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:43.821195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.821900 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.821906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:43.822079 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:43.822716 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:43.822734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:43.825578 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
W20260927 22:41:43.826174 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:43.817626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:57941 (local address 127.1.53.126:45165)
0927 22:41:43.817872 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:43.817882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:43.818154 (+ 272us) server_negotiation.cc:407] Connection header received
0927 22:41:43.818362 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:43.818366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:43.818427 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:43.818527 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:43.819332 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.820055 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.821329 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:43.821757 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:43.821888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:43.825771 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.65:57941: BlockingRecv error: recv got EOF from 127.1.53.65:57941 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:41:44 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:41:44 dist-test-slave-pp80 krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.53.66@KRBTEST.COM for oryx/127.1.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:41:44.286908 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:44.277721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45165 (local address 127.1.53.66:49755)
0927 22:41:44.278200 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:44.278217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:44.278325 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:44.278613 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:44.278617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:44.279005 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:44.279186 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:44.279194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.280347 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.280352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:44.281390 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:44.281403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.282443 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.282448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:44.282582 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:44.283259 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:44.283278 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:44.286546 (+ 3268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.53.126:45165: Server oryx/127.1.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":284,"threads_started":1}
W20260927 22:41:44.286967 10223 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:44.277831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:49755 (local address 127.1.53.126:45165)
0927 22:41:44.278083 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:44.278089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:44.278296 (+ 207us) server_negotiation.cc:407] Connection header received
0927 22:41:44.278430 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:44.278433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:44.278495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:44.278574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:44.279313 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.280088 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.281579 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.282290 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.282391 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:44.286733 (+ 4342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.53.66:49755: BlockingRecv error: recv got EOF from 127.1.53.66:49755 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:41:44.564019 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10074
I20260927 22:41:44.576848 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:44.565280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.1.53.65:39517)
0927 22:41:44.565851 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:44.565859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:44.565908 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:41:44.565976 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:44.565984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:44.566240 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:44.566408 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:44.567885 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.569115 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.570355 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.570992 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.571115 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:44.573618 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:44.573628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:44.573644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:44.573693 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:44.576347 (+ 2654us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:41:44.576622 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35464: Request ticket server kudu/127.1.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":256,"threads_started":1}
W20260927 22:41:44.577028 10534 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35464: Request ticket server kudu/127.1.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:41:44.577105 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10074: Not authorized: Client connection negotiation failed: client connection to 127.1.53.65:39517: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:41:44.577343 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10074
I20260927 22:41:44.594101 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10225
I20260927 22:41:44.606055 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:44.595309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.1.53.66:33177)
0927 22:41:44.595884 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:44.595891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:44.595921 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:41:44.595980 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:44.595987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:44.596208 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:44.596464 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:44.597668 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.598731 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.599869 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.600563 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.600659 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:44.602991 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:44.603002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:44.603020 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:44.603049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:44.605514 (+ 2465us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:41:44.605753 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41566: Request ticket server kudu/127.1.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":269,"threads_started":1}
W20260927 22:41:44.606155 10535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41566: Request ticket server kudu/127.1.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:41:44.606211 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10225: Not authorized: Client connection negotiation failed: client connection to 127.1.53.66:33177: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:41:44.606380 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10225
I20260927 22:41:44.623008 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10356
I20260927 22:41:44.635251 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:44.624140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.1.53.67:42681)
0927 22:41:44.624774 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:44.624781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:44.624808 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:41:44.624868 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:44.624874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:44.625122 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:44.625285 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:44.626664 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.627817 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.629090 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:44.629676 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:44.629779 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:44.632109 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:44.632118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:44.632132 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:44.632175 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:44.634768 (+ 2593us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:41:44.635022 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49194: Request ticket server kudu/127.1.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":347,"threads_started":1}
W20260927 22:41:44.635388 10536 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49194: Request ticket server kudu/127.1.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:41:44.635463 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10356: Not authorized: Client connection negotiation failed: client connection to 127.1.53.67:42681: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:41:44.635684 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10356
I20260927 22:41:44.651965 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10005
I20260927 22:41:44.653301 10067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:44.769538 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10005
2026-09-27T22:41:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21163 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:44Z Disabled control of system clock
I20260927 22:41:44.846416 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45177
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:41:45.177587 10541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:45.177901 10541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:45.177982 10541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:45.185142 10541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:45.185258 10541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:45.185295 10541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:45.185344 10541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:45.185389 10541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:45.194633 10541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:45177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:45.195955 10541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:45.197906 10541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:45.210669 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:45.210698 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:45.210695 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:45.211262 10541 server_base.cc:1034] running on GCE node
I20260927 22:41:45.212167 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:45.213636 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:45.214902 10541 hybrid_clock.cc:648] HybridClock initialized: now 1790548905214867 us; error 49 us; skew 500 ppm
I20260927 22:41:45.215309 10541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:45.217815 10541 webserver.cc:460] Webserver started at http://127.1.53.126:46861/ using document root <none> and password file <none>
I20260927 22:41:45.218604 10541 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:45.218695 10541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:45.219054 10541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:45.222043 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "0d67fe6fe85d4be4845e27c04eb0aa69"
format_stamp: "Formatted at 2026-09-27 22:41:45 on dist-test-slave-pp80"
server_key: "bb32a0c131317627493d7d5dce71abe9"
server_key_iv: "a27072e7adb3b10fc8203d74bbd14871"
server_key_version: "encryptionkey@0"
I20260927 22:41:45.222834 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "0d67fe6fe85d4be4845e27c04eb0aa69"
format_stamp: "Formatted at 2026-09-27 22:41:45 on dist-test-slave-pp80"
server_key: "bb32a0c131317627493d7d5dce71abe9"
server_key_iv: "a27072e7adb3b10fc8203d74bbd14871"
server_key_version: "encryptionkey@0"
I20260927 22:41:45.228125 10541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:41:45.231783 10555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:45.233397 10541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:45.233618 10541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "0d67fe6fe85d4be4845e27c04eb0aa69"
format_stamp: "Formatted at 2026-09-27 22:41:45 on dist-test-slave-pp80"
server_key: "bb32a0c131317627493d7d5dce71abe9"
server_key_iv: "a27072e7adb3b10fc8203d74bbd14871"
server_key_version: "encryptionkey@0"
I20260927 22:41:45.233853 10541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:45.258378 10541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:45.259296 10541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:45.259613 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:45.278486 10541 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:45177
I20260927 22:41:45.278534 10606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:45177 every 8 connection(s)
I20260927 22:41:45.279901 10541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:45.283907 10607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:45.288219 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10541
I20260927 22:41:45.288473 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:45.289125 1237 external_mini_cluster.cc:1428] Setting key 91188aeb1b1b5c0d63175777e45b81c3
I20260927 22:41:45.294605 10607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69: Bootstrap starting.
I20260927 22:41:45.298187 10607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:45.299360 10607 log.cc:824] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:45.302600 10607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69: No bootstrap required, opened a new log
I20260927 22:41:45.303150 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:45.291108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.1.53.126:45177)
0927 22:41:45.292122 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:45.292136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:45.292184 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:41:45.293868 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:45.293890 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:45.294338 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:45.294663 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:45.295333 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:45.296714 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:45.298633 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:45.299332 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:45.299473 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:45.300834 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:45.300846 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:45.300871 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:45.301402 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:45.301475 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:45.301504 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:45.301599 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:45.301820 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 22:41:45.302150 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":374,"threads_started":1}
I20260927 22:41:45.309876 10607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45177 } }
I20260927 22:41:45.310230 10607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:45.310353 10607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d67fe6fe85d4be4845e27c04eb0aa69, State: Initialized, Role: FOLLOWER
I20260927 22:41:45.311086 10607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45177 } }
I20260927 22:41:45.311336 10607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:45.311527 10607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:45.311693 10607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:45.313887 10607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45177 } }
I20260927 22:41:45.314339 10607 leader_election.cc:302] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d67fe6fe85d4be4845e27c04eb0aa69; no voters:
I20260927 22:41:45.314929 10607 leader_election.cc:288] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:45.315093 10612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:45.315760 10612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [term 1 LEADER]: Becoming Leader. State: Replica: 0d67fe6fe85d4be4845e27c04eb0aa69, State: Running, Role: LEADER
I20260927 22:41:45.316334 10612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45177 } }
I20260927 22:41:45.317183 10607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:45.319237 10613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45177 } } }
I20260927 22:41:45.319684 10613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:45.320216 10614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d67fe6fe85d4be4845e27c04eb0aa69. Latest consensus state: current_term: 1 leader_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67fe6fe85d4be4845e27c04eb0aa69" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45177 } } }
I20260927 22:41:45.320554 10614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:45.321277 10619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:45.326725 10619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:45.335870 10619 catalog_manager.cc:1347] Generated new cluster ID: aa503a9d74184850bbdb719f2b75e089
I20260927 22:41:45.336062 10619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:45.349541 10619 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:45.351042 10619 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:45.358834 10619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d67fe6fe85d4be4845e27c04eb0aa69: Generated new TSK 0
I20260927 22:41:45.359673 10619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:41:45.366803 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45177
--builtin_ntp_servers=127.1.53.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:41:45.703270 10631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:45.703570 10631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:45.703656 10631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:45.711169 10631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:45.711277 10631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:45.711383 10631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:45.721017 10631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:45.722326 10631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:45.724143 10631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:45.739322 10639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:45.739248 10637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:45.739928 10636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:45.740657 10631 server_base.cc:1034] running on GCE node
I20260927 22:41:45.741276 10631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:45.742234 10631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:45.743433 10631 hybrid_clock.cc:648] HybridClock initialized: now 1790548905743382 us; error 41 us; skew 500 ppm
I20260927 22:41:45.743772 10631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:45.746367 10631 webserver.cc:460] Webserver started at http://127.1.53.65:41993/ using document root <none> and password file <none>
I20260927 22:41:45.747092 10631 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:45.747177 10631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:45.747536 10631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:45.750491 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "dfc31254dc85412f920144ae2d539507"
format_stamp: "Formatted at 2026-09-27 22:41:45 on dist-test-slave-pp80"
server_key: "d6c379011c7af13849358e3b1608651e"
server_key_iv: "565432fc5ee0dc35744a08145e1ebdd2"
server_key_version: "encryptionkey@0"
I20260927 22:41:45.751276 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "dfc31254dc85412f920144ae2d539507"
format_stamp: "Formatted at 2026-09-27 22:41:45 on dist-test-slave-pp80"
server_key: "d6c379011c7af13849358e3b1608651e"
server_key_iv: "565432fc5ee0dc35744a08145e1ebdd2"
server_key_version: "encryptionkey@0"
I20260927 22:41:45.756567 10631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:41:45.759891 10645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:45.761330 10631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:45.761509 10631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "dfc31254dc85412f920144ae2d539507"
format_stamp: "Formatted at 2026-09-27 22:41:45 on dist-test-slave-pp80"
server_key: "d6c379011c7af13849358e3b1608651e"
server_key_iv: "565432fc5ee0dc35744a08145e1ebdd2"
server_key_version: "encryptionkey@0"
I20260927 22:41:45.761729 10631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:45.773320 10631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:45.774139 10631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:45.774427 10631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:45.775343 10631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:45.776877 10631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:45.776996 10631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:45.777122 10631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:45.777212 10631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:45.810352 10631 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:39601
I20260927 22:41:45.810403 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:39601 every 8 connection(s)
I20260927 22:41:45.811694 10631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:45.812023 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10631
I20260927 22:41:45.812222 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:45.813129 1237 external_mini_cluster.cc:1428] Setting key fce9532b3650db12631fa4113c224f34
I20260927 22:41:45.816431 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45177
--builtin_ntp_servers=127.1.53.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:41:45.826550 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:45.814640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:53229 (local address 127.1.53.126:45177)
0927 22:41:45.815134 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:45.815140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:45.816792 (+ 1652us) server_negotiation.cc:407] Connection header received
0927 22:41:45.818807 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:45.818813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:45.818914 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:45.819029 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:45.820141 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:45.821112 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:45.823259 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:45.824008 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:45.824159 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:45.825573 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:45.825582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:45.825590 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:45.825618 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:45.825669 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:45.825687 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:45.825788 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:45.826014 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 22:41:45.826121 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":288,"threads_started":1}
I20260927 22:41:45.827502 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:45.815426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45177 (local address 127.1.53.65:53229)
0927 22:41:45.816630 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:45.816681 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:45.818501 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:45.819282 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:45.819290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:45.819406 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:45.819972 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:45.819988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:45.821310 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:45.821319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:45.823108 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:45.823121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:45.824219 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:45.824224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:45.824510 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:45.825247 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:41:45.825249 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:45.825296 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:41:45.825376 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:45.825789 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:45.825797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:45.825802 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:41:45.825966 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:41:45.826276 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":330,"threads_started":1}
I20260927 22:41:45.829213 10758 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45177
I20260927 22:41:45.829519 10758 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:45.830269 10758 heartbeater.cc:511] Master 127.1.53.126:45177 requested a full tablet report, sending...
I20260927 22:41:45.832535 10572 ts_manager.cc:194] Registered new tserver with Master: dfc31254dc85412f920144ae2d539507 (127.1.53.65:39601)
I20260927 22:41:45.834970 10572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.65:53229
W20260927 22:41:46.141351 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:46.141624 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:46.141706 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:46.149171 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:46.149273 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:46.149376 10762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:46.158365 10762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:46.159693 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:46.161486 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:46.174404 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:46.174489 10767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:46.174466 10768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:46.175247 10762 server_base.cc:1034] running on GCE node
I20260927 22:41:46.175813 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:46.176802 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:46.178025 10762 hybrid_clock.cc:648] HybridClock initialized: now 1790548906177992 us; error 26 us; skew 500 ppm
I20260927 22:41:46.178344 10762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:46.180733 10762 webserver.cc:460] Webserver started at http://127.1.53.66:43771/ using document root <none> and password file <none>
I20260927 22:41:46.181469 10762 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:46.181556 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:46.181900 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:46.184727 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "b7ba2b827ac24e07943bb6a1bf292861"
format_stamp: "Formatted at 2026-09-27 22:41:46 on dist-test-slave-pp80"
server_key: "2ec50a5df6702c55c141092427e2ad57"
server_key_iv: "e2efcf0c1e20c30e8330a1cebe729c00"
server_key_version: "encryptionkey@0"
I20260927 22:41:46.185513 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ba2b827ac24e07943bb6a1bf292861"
format_stamp: "Formatted at 2026-09-27 22:41:46 on dist-test-slave-pp80"
server_key: "2ec50a5df6702c55c141092427e2ad57"
server_key_iv: "e2efcf0c1e20c30e8330a1cebe729c00"
server_key_version: "encryptionkey@0"
I20260927 22:41:46.190673 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:41:46.194244 10776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:46.195647 10762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:41:46.195847 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "b7ba2b827ac24e07943bb6a1bf292861"
format_stamp: "Formatted at 2026-09-27 22:41:46 on dist-test-slave-pp80"
server_key: "2ec50a5df6702c55c141092427e2ad57"
server_key_iv: "e2efcf0c1e20c30e8330a1cebe729c00"
server_key_version: "encryptionkey@0"
I20260927 22:41:46.196065 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:46.228315 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:46.229214 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:46.229506 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:46.230484 10762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:46.232009 10762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:46.232095 10762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:46.232228 10762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:46.232331 10762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:41:46.273931 10762 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:36965
I20260927 22:41:46.274009 10888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:36965 every 8 connection(s)
I20260927 22:41:46.275148 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:46.281316 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10762
I20260927 22:41:46.281464 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:46.281980 1237 external_mini_cluster.cc:1428] Setting key 04ef2077dc5a067feb6b230e0dc8877d
I20260927 22:41:46.284929 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45177
--builtin_ntp_servers=127.1.53.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:41:46.288486 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.277709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:47971 (local address 127.1.53.126:45177)
0927 22:41:46.277911 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:46.277917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:46.279646 (+ 1729us) server_negotiation.cc:407] Connection header received
0927 22:41:46.281445 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:46.281451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:46.281550 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:46.281651 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:46.282900 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.283993 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.285735 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.286172 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.286306 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.287633 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:46.287640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:46.287645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:46.287665 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:46.287701 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:46.287713 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:46.287805 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:46.288037 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:41:46.288126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:41:46.289512 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.278447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45177 (local address 127.1.53.66:47971)
0927 22:41:46.279483 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:46.279528 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:46.281181 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:46.281880 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:46.281888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:46.281981 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:46.282669 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:46.282695 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.284165 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.284171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:46.285596 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:46.285607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.286383 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.286389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:46.286608 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.287336 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:41:46.287339 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:46.287377 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:41:46.287447 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:46.287803 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:46.287815 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:46.287826 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:41:46.288015 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 22:41:46.288412 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":369,"threads_started":1}
I20260927 22:41:46.292342 10889 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45177
I20260927 22:41:46.292614 10889 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:46.293510 10889 heartbeater.cc:511] Master 127.1.53.126:45177 requested a full tablet report, sending...
I20260927 22:41:46.294799 10572 ts_manager.cc:194] Registered new tserver with Master: b7ba2b827ac24e07943bb6a1bf292861 (127.1.53.66:36965)
I20260927 22:41:46.295594 10572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.66:47971
W20260927 22:41:46.614280 10892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:46.614558 10892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:46.614640 10892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:46.622205 10892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:46.622313 10892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:46.622419 10892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:46.631721 10892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:46.632985 10892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:46.634646 10892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:46.647397 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:46.647439 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:46.647377 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:46.648377 10892 server_base.cc:1034] running on GCE node
I20260927 22:41:46.648983 10892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:46.650038 10892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:46.651268 10892 hybrid_clock.cc:648] HybridClock initialized: now 1790548906651199 us; error 72 us; skew 500 ppm
I20260927 22:41:46.651597 10892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:46.653867 10892 webserver.cc:460] Webserver started at http://127.1.53.67:34781/ using document root <none> and password file <none>
I20260927 22:41:46.654583 10892 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:46.654718 10892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:46.655041 10892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:46.657892 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "06540bfa19b04e59bbaacc883c5b696f"
format_stamp: "Formatted at 2026-09-27 22:41:46 on dist-test-slave-pp80"
server_key: "70e267cfe125ea57da7e8a1db28456e8"
server_key_iv: "8a47a5e926c0c886c1272c781044d134"
server_key_version: "encryptionkey@0"
I20260927 22:41:46.658746 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "06540bfa19b04e59bbaacc883c5b696f"
format_stamp: "Formatted at 2026-09-27 22:41:46 on dist-test-slave-pp80"
server_key: "70e267cfe125ea57da7e8a1db28456e8"
server_key_iv: "8a47a5e926c0c886c1272c781044d134"
server_key_version: "encryptionkey@0"
I20260927 22:41:46.663825 10892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:41:46.667430 10906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:46.668830 10892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:46.669005 10892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "06540bfa19b04e59bbaacc883c5b696f"
format_stamp: "Formatted at 2026-09-27 22:41:46 on dist-test-slave-pp80"
server_key: "70e267cfe125ea57da7e8a1db28456e8"
server_key_iv: "8a47a5e926c0c886c1272c781044d134"
server_key_version: "encryptionkey@0"
I20260927 22:41:46.669212 10892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:46.682600 10892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:46.683457 10892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:46.683730 10892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:46.684725 10892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:46.686146 10892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:46.686266 10892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:46.686410 10892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:46.686499 10892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:46.720929 10892 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:45889
I20260927 22:41:46.720978 11018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:45889 every 8 connection(s)
I20260927 22:41:46.722340 10892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:46.732527 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 10892
I20260927 22:41:46.732707 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:46.733151 1237 external_mini_cluster.cc:1428] Setting key 5ac84de5cb0fc07df054a03798ae7cc2
I20260927 22:41:46.734793 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.724916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:42279 (local address 127.1.53.126:45177)
0927 22:41:46.725090 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:46.725095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:46.726535 (+ 1440us) server_negotiation.cc:407] Connection header received
0927 22:41:46.728291 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:46.728297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:46.728390 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:46.728502 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:46.729546 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.730278 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.731792 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.732225 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.732421 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.734015 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:46.734022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:46.734026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:46.734045 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:46.734081 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:46.734096 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:46.734204 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:46.734469 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:41:46.734566 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:41:46.736071 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.725478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45177 (local address 127.1.53.67:42279)
0927 22:41:46.726361 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:46.726402 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:46.727994 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:46.728736 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:46.728745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:46.728831 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:46.729380 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:46.729396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.730423 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.730430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:46.731648 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:46.731659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.732527 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.732535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:46.732837 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.733644 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:41:46.733647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:46.733717 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:41:46.733819 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:46.734214 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:46.734226 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:46.734235 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:41:46.734461 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 22:41:46.734880 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":281,"threads_started":1}
I20260927 22:41:46.737587 11019 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45177
I20260927 22:41:46.737896 11019 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:46.738682 11019 heartbeater.cc:511] Master 127.1.53.126:45177 requested a full tablet report, sending...
I20260927 22:41:46.740054 10572 ts_manager.cc:194] Registered new tserver with Master: 06540bfa19b04e59bbaacc883c5b696f (127.1.53.67:45889)
I20260927 22:41:46.740844 10572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.67:42279
I20260927 22:41:46.748721 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:41:46.754688 10760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:46.753256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.1.53.126:45177)
0927 22:41:46.753453 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:46.753458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:46.753933 (+ 475us) server_negotiation.cc:407] Connection header received
0927 22:41:46.754055 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:46.754059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:46.754130 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:46.754219 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:46.754501 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60404: BlockingRecv error: recv got EOF from 127.0.0.1:60404 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:41:46.755331 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10631
I20260927 22:41:46.763226 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.756308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.1.53.65:39601)
0927 22:41:46.756875 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:46.756884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:46.756918 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:41:46.756970 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:46.756975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:46.757189 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:46.757356 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:46.757740 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.758701 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.759940 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.760579 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.760717 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.761802 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:46.761810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:46.761824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:46.762453 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:46.762501 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:46.762520 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:46.762646 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:46.762771 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 22:41:46.762897 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":225,"threads_started":1}
I20260927 22:41:46.764626 10753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:46.847618 10758 heartbeater.cc:503] Master 127.1.53.126:45177 was elected leader, sending a full tablet report...
I20260927 22:41:46.848831 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10631
W20260927 22:41:46.865173 10558 connection.cc:535] server connection from 127.1.53.65:53229 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:41:46.865576 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10762
I20260927 22:41:46.874009 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.866573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.1.53.66:36965)
0927 22:41:46.867126 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:46.867134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:46.867161 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:41:46.867227 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:46.867233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:46.867464 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:46.867590 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:46.867983 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.869094 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.870290 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.870843 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.870946 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.872211 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:46.872219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:46.872304 (+ 85us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:46.873055 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:46.873101 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:46.873121 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:46.873236 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:46.873422 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 22:41:46.873534 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":263,"threads_started":1}
I20260927 22:41:46.875245 10884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:46.959100 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10762
I20260927 22:41:46.975488 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10892
I20260927 22:41:46.983879 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:46.976498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.1.53.67:45889)
0927 22:41:46.977216 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:46.977223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:46.977253 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:41:46.977317 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:46.977323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:46.977545 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:46.977683 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:46.978127 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.979162 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.980573 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:46.981144 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:46.981244 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:46.982404 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:46.982412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:46.982425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:46.983027 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:46.983071 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:46.983094 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:46.983180 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:46.983361 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 22:41:46.983522 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":436,"threads_started":1}
I20260927 22:41:46.985261 11014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:47.068691 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10892
I20260927 22:41:47.084365 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 10541
I20260927 22:41:47.085702 10602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:47.187438 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 10541
2026-09-27T22:41:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2412 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:47Z Disabled control of system clock
I20260927 22:41:47.260649 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46167
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:41:47.593772 11038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:47.594081 11038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:47.594162 11038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:47.601307 11038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:47.601423 11038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:47.601460 11038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:47.601508 11038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:47.601552 11038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:47.611029 11038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:47.612461 11038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:47.614161 11038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:47.626268 11044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:47.626327 11046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:47.626294 11043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:47.627743 11038 server_base.cc:1034] running on GCE node
I20260927 22:41:47.628954 11038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:47.630914 11038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:47.632179 11038 hybrid_clock.cc:648] HybridClock initialized: now 1790548907632112 us; error 53 us; skew 500 ppm
I20260927 22:41:47.632647 11038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:47.635732 11038 webserver.cc:460] Webserver started at http://127.1.53.126:32787/ using document root <none> and password file <none>
I20260927 22:41:47.636683 11038 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:47.636837 11038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:47.637348 11038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:47.642151 11038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "cfa3916a86d74f6f9650b16d2845c01e"
format_stamp: "Formatted at 2026-09-27 22:41:47 on dist-test-slave-pp80"
server_key: "8c20ce7d42745b53fcc7fc9e265cf338"
server_key_iv: "efef2ae421308735cd4fdc18992e8408"
server_key_version: "encryptionkey@0"
I20260927 22:41:47.643141 11038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "cfa3916a86d74f6f9650b16d2845c01e"
format_stamp: "Formatted at 2026-09-27 22:41:47 on dist-test-slave-pp80"
server_key: "8c20ce7d42745b53fcc7fc9e265cf338"
server_key_iv: "efef2ae421308735cd4fdc18992e8408"
server_key_version: "encryptionkey@0"
I20260927 22:41:47.649463 11038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 22:41:47.653057 11052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:47.654450 11038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:41:47.654635 11038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "cfa3916a86d74f6f9650b16d2845c01e"
format_stamp: "Formatted at 2026-09-27 22:41:47 on dist-test-slave-pp80"
server_key: "8c20ce7d42745b53fcc7fc9e265cf338"
server_key_iv: "efef2ae421308735cd4fdc18992e8408"
server_key_version: "encryptionkey@0"
I20260927 22:41:47.654852 11038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:47.660666 11038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:47.661419 11038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:47.661710 11038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:47.679563 11038 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:46167
I20260927 22:41:47.679595 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:46167 every 8 connection(s)
I20260927 22:41:47.680971 11038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:47.683637 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11038
I20260927 22:41:47.683856 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:47.684548 1237 external_mini_cluster.cc:1428] Setting key a60ae457685e7179d6edd6b40c76d912
I20260927 22:41:47.684731 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:47.692842 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:47.686488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.1.53.126:46167)
0927 22:41:47.687487 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:47.687500 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:47.687539 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:41:47.689307 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:47.689334 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:47.689742 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:47.690128 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:47.691187 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:47.691193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:47.691205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:47.691221 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:47.691279 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:47.691297 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:47.691468 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:47.691580 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 22:41:47.691918 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":381,"threads_started":1}
I20260927 22:41:47.697517 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e: Bootstrap starting.
I20260927 22:41:47.700353 11104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:47.701411 11104 log.cc:824] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:47.705798 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e: No bootstrap required, opened a new log
I20260927 22:41:47.714287 11104 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3916a86d74f6f9650b16d2845c01e" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46167 } }
I20260927 22:41:47.714682 11104 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:47.714783 11104 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa3916a86d74f6f9650b16d2845c01e, State: Initialized, Role: FOLLOWER
I20260927 22:41:47.715381 11104 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3916a86d74f6f9650b16d2845c01e" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46167 } }
I20260927 22:41:47.715602 11104 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:47.715727 11104 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:47.715900 11104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:47.718276 11104 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3916a86d74f6f9650b16d2845c01e" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46167 } }
I20260927 22:41:47.718799 11104 leader_election.cc:302] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfa3916a86d74f6f9650b16d2845c01e; no voters:
I20260927 22:41:47.719451 11104 leader_election.cc:288] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:47.719626 11109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:47.720350 11109 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [term 1 LEADER]: Becoming Leader. State: Replica: cfa3916a86d74f6f9650b16d2845c01e, State: Running, Role: LEADER
I20260927 22:41:47.720981 11109 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3916a86d74f6f9650b16d2845c01e" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46167 } }
I20260927 22:41:47.721540 11104 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:47.723717 11110 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfa3916a86d74f6f9650b16d2845c01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3916a86d74f6f9650b16d2845c01e" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46167 } } }
I20260927 22:41:47.724105 11110 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:47.724606 11111 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa3916a86d74f6f9650b16d2845c01e. Latest consensus state: current_term: 1 leader_uuid: "cfa3916a86d74f6f9650b16d2845c01e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa3916a86d74f6f9650b16d2845c01e" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46167 } } }
I20260927 22:41:47.725001 11111 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:47.728504 11118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:47.732640 11118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:47.740685 11118 catalog_manager.cc:1347] Generated new cluster ID: 6b1a6d3b4f4549eaa04de8efcf1bc4f1
I20260927 22:41:47.740803 11118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:47.754639 11118 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:47.755110 11118 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:47.764658 11118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfa3916a86d74f6f9650b16d2845c01e: Generated new TSK 0
I20260927 22:41:47.765702 11118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:41:47.772828 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46167
--builtin_ntp_servers=127.1.53.84:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:41:48.101595 11129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:48.101931 11129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:48.102113 11129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:48.109308 11129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:48.109459 11129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:48.109648 11129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:48.118594 11129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:48.119894 11129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:48.121665 11129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:48.134446 11134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:48.134645 11135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:48.135020 11137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:48.139173 11129 server_base.cc:1034] running on GCE node
I20260927 22:41:48.139794 11129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:48.140830 11129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:48.142042 11129 hybrid_clock.cc:648] HybridClock initialized: now 1790548908141994 us; error 40 us; skew 500 ppm
I20260927 22:41:48.142395 11129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:48.144727 11129 webserver.cc:460] Webserver started at http://127.1.53.65:39381/ using document root <none> and password file <none>
I20260927 22:41:48.145363 11129 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:48.145457 11129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:48.145773 11129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:48.148697 11129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "7892875412c944e3a15487ebb49c3727"
format_stamp: "Formatted at 2026-09-27 22:41:48 on dist-test-slave-pp80"
server_key: "5f671bd4a02b7a074b2b4804848ed6c4"
server_key_iv: "4954a6ca935b6bea76871f6030719b68"
server_key_version: "encryptionkey@0"
I20260927 22:41:48.149436 11129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "7892875412c944e3a15487ebb49c3727"
format_stamp: "Formatted at 2026-09-27 22:41:48 on dist-test-slave-pp80"
server_key: "5f671bd4a02b7a074b2b4804848ed6c4"
server_key_iv: "4954a6ca935b6bea76871f6030719b68"
server_key_version: "encryptionkey@0"
I20260927 22:41:48.154657 11129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:41:48.158036 11143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:48.159504 11129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:41:48.159631 11129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "7892875412c944e3a15487ebb49c3727"
format_stamp: "Formatted at 2026-09-27 22:41:48 on dist-test-slave-pp80"
server_key: "5f671bd4a02b7a074b2b4804848ed6c4"
server_key_iv: "4954a6ca935b6bea76871f6030719b68"
server_key_version: "encryptionkey@0"
I20260927 22:41:48.159812 11129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:48.165309 11129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:48.166067 11129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:48.166312 11129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:48.167172 11129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:48.168596 11129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:48.168681 11129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:48.168753 11129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:48.168807 11129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:48.202076 11129 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:45643
I20260927 22:41:48.202149 11255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:45643 every 8 connection(s)
I20260927 22:41:48.203408 11129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:48.207232 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11129
I20260927 22:41:48.207473 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:48.208176 1237 external_mini_cluster.cc:1428] Setting key 754d31fe8a01502d6101622eaea4fcee
I20260927 22:41:48.211201 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46167
--builtin_ntp_servers=127.1.53.84:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:41:48.212182 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:48.206253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:37585 (local address 127.1.53.126:46167)
0927 22:41:48.206685 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:48.206690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:48.208157 (+ 1467us) server_negotiation.cc:407] Connection header received
0927 22:41:48.209846 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:48.209852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:48.209916 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:48.210045 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:48.211508 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:48.211512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:48.211515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:48.211530 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:48.211569 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:48.211580 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:48.211666 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:48.211938 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:41:48.211984 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":231,"threads_started":1}
I20260927 22:41:48.213732 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:48.206890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46167 (local address 127.1.53.65:37585)
0927 22:41:48.207965 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:48.208010 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:48.209577 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:48.210286 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:48.210294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:48.210390 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:48.211145 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:41:48.211148 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:48.211191 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:41:48.211274 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:48.211690 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:48.211706 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:48.211719 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:41:48.212033 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 22:41:48.212379 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":338,"threads_started":1}
I20260927 22:41:48.215464 11256 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46167
I20260927 22:41:48.215816 11256 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:48.216645 11256 heartbeater.cc:511] Master 127.1.53.126:46167 requested a full tablet report, sending...
I20260927 22:41:48.219226 11069 ts_manager.cc:194] Registered new tserver with Master: 7892875412c944e3a15487ebb49c3727 (127.1.53.65:45643)
W20260927 22:41:48.538357 11260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:48.538641 11260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:48.538741 11260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:48.546773 11260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:48.546892 11260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:48.547012 11260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:48.557093 11260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:48.558516 11260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:48.560417 11260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:48.574203 11265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:48.574244 11268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:48.574815 11266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:48.575721 11260 server_base.cc:1034] running on GCE node
I20260927 22:41:48.576371 11260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:48.577265 11260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:48.578503 11260 hybrid_clock.cc:648] HybridClock initialized: now 1790548908578465 us; error 52 us; skew 500 ppm
I20260927 22:41:48.578830 11260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:48.581161 11260 webserver.cc:460] Webserver started at http://127.1.53.66:37083/ using document root <none> and password file <none>
I20260927 22:41:48.581880 11260 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:48.582012 11260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:48.582343 11260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:48.585328 11260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "e850306ab2c74b368eea6398087fe1fd"
format_stamp: "Formatted at 2026-09-27 22:41:48 on dist-test-slave-pp80"
server_key: "176a712513bc406b9eaa2b64f5a22c26"
server_key_iv: "67ac2c36874b0d31bae1e7e238cff0d8"
server_key_version: "encryptionkey@0"
I20260927 22:41:48.586145 11260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "e850306ab2c74b368eea6398087fe1fd"
format_stamp: "Formatted at 2026-09-27 22:41:48 on dist-test-slave-pp80"
server_key: "176a712513bc406b9eaa2b64f5a22c26"
server_key_iv: "67ac2c36874b0d31bae1e7e238cff0d8"
server_key_version: "encryptionkey@0"
I20260927 22:41:48.591176 11260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:41:48.594663 11274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:48.596057 11260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:41:48.596186 11260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "e850306ab2c74b368eea6398087fe1fd"
format_stamp: "Formatted at 2026-09-27 22:41:48 on dist-test-slave-pp80"
server_key: "176a712513bc406b9eaa2b64f5a22c26"
server_key_iv: "67ac2c36874b0d31bae1e7e238cff0d8"
server_key_version: "encryptionkey@0"
I20260927 22:41:48.596429 11260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:48.602005 11260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:48.602697 11260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:48.602910 11260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:48.603724 11260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:48.605100 11260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:48.605183 11260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:48.605260 11260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:48.605314 11260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:48.638578 11260 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:45279
I20260927 22:41:48.638664 11386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:45279 every 8 connection(s)
I20260927 22:41:48.639843 11260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:48.646432 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11260
I20260927 22:41:48.646641 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:48.647078 1237 external_mini_cluster.cc:1428] Setting key 3d405b0f39966a41b480014edf88060c
I20260927 22:41:48.648012 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:48.642403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:41321 (local address 127.1.53.126:46167)
0927 22:41:48.642614 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:48.642620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:48.644036 (+ 1416us) server_negotiation.cc:407] Connection header received
0927 22:41:48.645681 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:48.645686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:48.645752 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:48.645903 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:48.647221 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:48.647226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:48.647231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:48.647250 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:48.647291 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:48.647303 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:48.647430 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:48.647617 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 22:41:48.647683 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:41:48.649052 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:48.642972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46167 (local address 127.1.53.66:41321)
0927 22:41:48.643881 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:48.643924 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:48.645424 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:48.646084 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:48.646111 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:48.646197 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:48.646941 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:41:48.646943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:48.646978 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:41:48.647058 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:48.647402 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:48.647411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:48.647419 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:41:48.647700 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 22:41:48.647937 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260927 22:41:48.649896 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46167
--builtin_ntp_servers=127.1.53.84:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:41:48.650333 11387 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46167
I20260927 22:41:48.650626 11387 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:48.651293 11387 heartbeater.cc:511] Master 127.1.53.126:46167 requested a full tablet report, sending...
I20260927 22:41:48.652738 11069 ts_manager.cc:194] Registered new tserver with Master: e850306ab2c74b368eea6398087fe1fd (127.1.53.66:45279)
W20260927 22:41:48.981209 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:48.981477 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:48.981564 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:48.988740 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:48.988839 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:48.989001 11390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:48.997830 11390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:48.999116 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:49.000900 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:49.013283 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:49.013284 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:49.013893 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:49.014973 11390 server_base.cc:1034] running on GCE node
I20260927 22:41:49.015515 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:49.016439 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:49.017637 11390 hybrid_clock.cc:648] HybridClock initialized: now 1790548909017589 us; error 38 us; skew 500 ppm
I20260927 22:41:49.017944 11390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:41:49.027058 11390 webserver.cc:460] Webserver started at http://127.1.53.67:35331/ using document root <none> and password file <none>
I20260927 22:41:49.027748 11390 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:49.027858 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:49.028311 11390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:49.031091 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "a86e409ef54c4908bca4dcb6538300dc"
format_stamp: "Formatted at 2026-09-27 22:41:49 on dist-test-slave-pp80"
server_key: "0b3ce3743d8d49f513879c0bac76f53e"
server_key_iv: "896ee50afdc6866ba535a2902df498fe"
server_key_version: "encryptionkey@0"
I20260927 22:41:49.031868 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "a86e409ef54c4908bca4dcb6538300dc"
format_stamp: "Formatted at 2026-09-27 22:41:49 on dist-test-slave-pp80"
server_key: "0b3ce3743d8d49f513879c0bac76f53e"
server_key_iv: "896ee50afdc6866ba535a2902df498fe"
server_key_version: "encryptionkey@0"
I20260927 22:41:49.036898 11390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:41:49.040315 11404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:49.041651 11390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:41:49.041821 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "a86e409ef54c4908bca4dcb6538300dc"
format_stamp: "Formatted at 2026-09-27 22:41:49 on dist-test-slave-pp80"
server_key: "0b3ce3743d8d49f513879c0bac76f53e"
server_key_iv: "896ee50afdc6866ba535a2902df498fe"
server_key_version: "encryptionkey@0"
I20260927 22:41:49.042045 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:49.047556 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:49.048357 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:49.048619 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:49.049502 11390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:49.050836 11390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:49.050928 11390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:49.051052 11390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:49.051149 11390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:49.084745 11390 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:43689
I20260927 22:41:49.084833 11516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:43689 every 8 connection(s)
I20260927 22:41:49.086079 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:49.093189 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11390
I20260927 22:41:49.093402 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:49.094071 1237 external_mini_cluster.cc:1428] Setting key 2116c95e17a763df39adb621865cdf14
I20260927 22:41:49.094977 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:49.088716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:42621 (local address 127.1.53.126:46167)
0927 22:41:49.088891 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:49.088896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:49.090444 (+ 1548us) server_negotiation.cc:407] Connection header received
0927 22:41:49.092290 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:49.092297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:49.092378 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:49.092504 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:49.094161 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:49.094168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:49.094174 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:49.094197 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:49.094234 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:49.094247 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:49.094352 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:49.094590 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:41:49.094667 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:41:49.096093 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:49.089263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46167 (local address 127.1.53.67:42621)
0927 22:41:49.090261 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:49.090299 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:49.091987 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:49.092775 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:49.092789 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:49.092925 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:49.093784 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:41:49.093788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:49.093842 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:41:49.093961 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:49.094345 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:49.094357 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:49.094364 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:41:49.094673 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 22:41:49.094918 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":326,"threads_started":1}
I20260927 22:41:49.097644 11517 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46167
I20260927 22:41:49.097930 11517 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:49.098490 11517 heartbeater.cc:511] Master 127.1.53.126:46167 requested a full tablet report, sending...
I20260927 22:41:49.099941 11068 ts_manager.cc:194] Registered new tserver with Master: a86e409ef54c4908bca4dcb6538300dc (127.1.53.67:43689)
I20260927 22:41:49.109633 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:41:49.115576 11258 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:41:49.114111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.1.53.126:46167)
0927 22:41:49.114308 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:49.114312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:49.114831 (+ 519us) server_negotiation.cc:407] Connection header received
0927 22:41:49.114974 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:49.114978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:49.115028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:49.115116 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:49.115391 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39516: BlockingRecv error: recv got EOF from 127.0.0.1:39516 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:41:49.116526 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11129
I20260927 22:41:49.120003 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:49.117469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.1.53.65:45643)
0927 22:41:49.117997 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:49.118004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:49.118028 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:41:49.118081 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:49.118087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:49.118300 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:49.118424 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:49.119370 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:49.119377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:49.119390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:49.119409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:49.119450 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:49.119464 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:49.119564 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:49.119645 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 22:41:49.119717 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
I20260927 22:41:49.121199 11251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:49.202702 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11129
I20260927 22:41:49.219002 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11260
I20260927 22:41:49.222759 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:49.220016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.1.53.66:45279)
0927 22:41:49.220620 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:49.220627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:49.220655 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:41:49.220714 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:49.220720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:49.220937 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:49.221068 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:49.222053 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:49.222060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:49.222071 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:49.222081 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:49.222122 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:49.222138 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:49.222241 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:49.222385 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:41:49.222509 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":344,"threads_started":1}
I20260927 22:41:49.224016 11382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:49.304579 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11260
I20260927 22:41:49.321584 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11390
I20260927 22:41:49.325910 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:49.322699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.1.53.67:43689)
0927 22:41:49.323399 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:49.323407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:49.323437 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:41:49.323499 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:49.323505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:49.323727 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:49.323898 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:49.325145 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:49.325151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:49.325162 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:41:49.325178 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:49.325215 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:41:49.325228 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:49.325422 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:49.325496 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:41:49.325595 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":375,"threads_started":1}
I20260927 22:41:49.327155 11512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:49.408111 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11390
I20260927 22:41:49.423689 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11038
I20260927 22:41:49.424998 11099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:49.524510 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11038
2026-09-27T22:41:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2336 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:49 dist-test-slave-pp80 krb5kdc[11535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:49 dist-test-slave-pp80 krb5kdc[11535](info): set up 2 sockets
Sep 27 22:41:49 dist-test-slave-pp80 krb5kdc[11535](info): commencing operation
krb5kdc: starting...
W20260927 22:41:50.645376 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:50 dist-test-slave-pp80 krb5kdc[11535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:41:50.884580 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37451
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:51.205881 11548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:51.206180 11548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:51.206269 11548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:51.213295 11548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:41:51.213397 11548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:51.213442 11548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:51.213491 11548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:41:51.213536 11548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:41:51.223603 11548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:51.224869 11548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:51.226085 11548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:51.238027 11553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:51.237998 11554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:51.238201 11556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:51.238714 11548 server_base.cc:1034] running on GCE node
I20260927 22:41:51.239480 11548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:51.240899 11548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:51.242128 11548 hybrid_clock.cc:648] HybridClock initialized: now 1790548911242084 us; error 38 us; skew 500 ppm
I20260927 22:41:51.242477 11548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:51 dist-test-slave-pp80 krb5kdc[11535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548911, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:51.245548 11548 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:41:51.246953 11548 webserver.cc:460] Webserver started at http://127.1.53.126:42657/ using document root <none> and password file <none>
I20260927 22:41:51.247579 11548 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:51.247673 11548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:51.247908 11548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:51.250801 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "d19978d4b58245749c55cfb06038e1f8"
format_stamp: "Formatted at 2026-09-27 22:41:51 on dist-test-slave-pp80"
server_key: "067336c9b540134dbe40bf1298bdb61a"
server_key_iv: "fbcb82d5a829da63c0777bb042e0fd39"
server_key_version: "encryptionkey@0"
I20260927 22:41:51.251500 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "d19978d4b58245749c55cfb06038e1f8"
format_stamp: "Formatted at 2026-09-27 22:41:51 on dist-test-slave-pp80"
server_key: "067336c9b540134dbe40bf1298bdb61a"
server_key_iv: "fbcb82d5a829da63c0777bb042e0fd39"
server_key_version: "encryptionkey@0"
I20260927 22:41:51.256717 11548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:41:51.260195 11563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:51.261627 11548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:41:51.261771 11548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "d19978d4b58245749c55cfb06038e1f8"
format_stamp: "Formatted at 2026-09-27 22:41:51 on dist-test-slave-pp80"
server_key: "067336c9b540134dbe40bf1298bdb61a"
server_key_iv: "fbcb82d5a829da63c0777bb042e0fd39"
server_key_version: "encryptionkey@0"
I20260927 22:41:51.261934 11548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:51.281512 11548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:51.286156 11548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:51.286388 11548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:51.305399 11548 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37451
I20260927 22:41:51.305441 11614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37451 every 8 connection(s)
I20260927 22:41:51.306769 11548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:41:51.307802 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11548
I20260927 22:41:51.308066 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:41:51.308911 1237 external_mini_cluster.cc:1428] Setting key 2c591ce39f6a3967946a9538b2979c30
I20260927 22:41:51.311909 11615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:41:51 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:51.326206 11615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8: Bootstrap starting.
I20260927 22:41:51.329730 11615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:51.331081 11615 log.cc:824] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:51.333379 11618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:51.311102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.1.53.126:37451)
0927 22:41:51.311901 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:51.311913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:51.311952 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:41:51.313535 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:51.313556 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:51.313905 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:51.314210 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:51.316164 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:51.317529 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:51.319447 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:51.320098 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:51.320294 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:51.325219 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:51.325227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:51.325242 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:51.325276 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:51.328810 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:51.329605 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:51.329620 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:51.329632 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:51.329814 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:51.330328 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:51.330332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:51.330334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:51.330860 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:51.331111 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:51.331636 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 22:41:51.332151 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":398,"threads_started":1}
I20260927 22:41:51.334431 11615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8: No bootstrap required, opened a new log
I20260927 22:41:51.341933 11615 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19978d4b58245749c55cfb06038e1f8" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37451 } }
I20260927 22:41:51.342280 11615 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:51.342396 11615 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19978d4b58245749c55cfb06038e1f8, State: Initialized, Role: FOLLOWER
I20260927 22:41:51.342963 11615 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19978d4b58245749c55cfb06038e1f8" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37451 } }
I20260927 22:41:51.343202 11615 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:41:51.343335 11615 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:41:51.343523 11615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:51.345999 11615 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19978d4b58245749c55cfb06038e1f8" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37451 } }
I20260927 22:41:51.346470 11615 leader_election.cc:302] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19978d4b58245749c55cfb06038e1f8; no voters:
I20260927 22:41:51.347030 11615 leader_election.cc:288] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:41:51.347157 11620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:51.347832 11620 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [term 1 LEADER]: Becoming Leader. State: Replica: d19978d4b58245749c55cfb06038e1f8, State: Running, Role: LEADER
I20260927 22:41:51.348407 11620 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19978d4b58245749c55cfb06038e1f8" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37451 } }
I20260927 22:41:51.349109 11615 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:41:51.351282 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19978d4b58245749c55cfb06038e1f8. Latest consensus state: current_term: 1 leader_uuid: "d19978d4b58245749c55cfb06038e1f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19978d4b58245749c55cfb06038e1f8" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37451 } } }
I20260927 22:41:51.351439 11621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19978d4b58245749c55cfb06038e1f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19978d4b58245749c55cfb06038e1f8" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37451 } } }
I20260927 22:41:51.351718 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:51.351734 11621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:41:51.353847 11628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:41:51.358820 11628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:41:51.368000 11628 catalog_manager.cc:1347] Generated new cluster ID: 31607d4cc7ad40368b5129970ec74f53
I20260927 22:41:51.368182 11628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:41:51.379195 11628 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:41:51.381075 11628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:41:51.398818 11628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8: Generated new TSK 0
I20260927 22:41:51.399724 11628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:41:51.464627 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37451
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:51.797609 11641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:51.797899 11641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:51.798002 11641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:51.805097 11641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:51.805203 11641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:51.805311 11641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:41:51.814822 11641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:51.816159 11641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:51.817479 11641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:51.830510 11646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:51.830578 11647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:51.830729 11649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:51.831446 11641 server_base.cc:1034] running on GCE node
I20260927 22:41:51.832062 11641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:51.833276 11641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:51.834528 11641 hybrid_clock.cc:648] HybridClock initialized: now 1790548911834485 us; error 33 us; skew 500 ppm
I20260927 22:41:51.834918 11641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:51 dist-test-slave-pp80 krb5kdc[11535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548911, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:51.838251 11641 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:41:51.839715 11641 webserver.cc:460] Webserver started at http://127.1.53.65:32777/ using document root <none> and password file <none>
I20260927 22:41:51.840497 11641 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:51.840624 11641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:51.840934 11641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:51.843830 11641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "928c08f00de04dfcbbf7cdd0ae667a1b"
format_stamp: "Formatted at 2026-09-27 22:41:51 on dist-test-slave-pp80"
server_key: "240e549840662309e8871bb0e0398c38"
server_key_iv: "151bb593a682198eecd31529004e6bab"
server_key_version: "encryptionkey@0"
I20260927 22:41:51.844632 11641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "928c08f00de04dfcbbf7cdd0ae667a1b"
format_stamp: "Formatted at 2026-09-27 22:41:51 on dist-test-slave-pp80"
server_key: "240e549840662309e8871bb0e0398c38"
server_key_iv: "151bb593a682198eecd31529004e6bab"
server_key_version: "encryptionkey@0"
I20260927 22:41:51.849879 11641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:41:51.853560 11656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:51.855095 11641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:41:51.855274 11641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "928c08f00de04dfcbbf7cdd0ae667a1b"
format_stamp: "Formatted at 2026-09-27 22:41:51 on dist-test-slave-pp80"
server_key: "240e549840662309e8871bb0e0398c38"
server_key_iv: "151bb593a682198eecd31529004e6bab"
server_key_version: "encryptionkey@0"
I20260927 22:41:51.855485 11641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:51.891943 11641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:51.896489 11641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:51.896795 11641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:51.897835 11641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:51.899415 11641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:51.899523 11641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:51.899676 11641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:51.899748 11641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:51.933605 11641 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:45595
I20260927 22:41:51.933665 11768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:45595 every 8 connection(s)
I20260927 22:41:51.934877 11641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:41:51.937975 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11641
I20260927 22:41:51.938223 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:41:51.939059 1237 external_mini_cluster.cc:1428] Setting key 0e247eb26a4c0923c2ad319aca13a612
Sep 27 22:41:51 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548911, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:51.958257 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:51.937642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:51819 (local address 127.1.53.126:37451)
0927 22:41:51.939019 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:51.939026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:51.939729 (+ 703us) server_negotiation.cc:407] Connection header received
0927 22:41:51.941882 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:51.941889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:51.941973 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:51.942091 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:51.943519 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:51.944662 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:51.946293 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:51.947011 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:51.947161 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:51.951310 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:51.951319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:51.951327 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:51.951381 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:51.954956 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:51.955694 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:51.955702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:51.955708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:51.955848 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:51.956265 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:51.956273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:51.956279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:51.956655 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:51.956808 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:51.957513 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 22:41:51.957767 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1238,"thread_start_us":1109,"threads_started":1}
I20260927 22:41:51.959111 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:51.938352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37451 (local address 127.1.53.65:51819)
0927 22:41:51.939543 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:51.939590 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:51.941605 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:51.942295 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:51.942302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:51.942799 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:51.943344 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:51.943361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:51.944855 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:51.944859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:51.946130 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:51.946142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:51.947223 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:51.947227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:51.947426 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:51.948168 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:51.948214 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:51.951091 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:51.955105 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:51.955116 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:51.955130 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:51.955572 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:51.955958 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:51.955962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:51.955964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:51.956136 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:51.956792 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:51.956798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:51.957122 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:41:51.957364 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 22:41:51.957746 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":665,"threads_started":1}
I20260927 22:41:51.960825 11769 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37451
I20260927 22:41:51.961179 11769 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:51.962118 11769 heartbeater.cc:511] Master 127.1.53.126:37451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:41:51.964588 11580 ts_manager.cc:194] Registered new tserver with Master: 928c08f00de04dfcbbf7cdd0ae667a1b (127.1.53.65:45595)
I20260927 22:41:51.966214 11580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:51819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:41:51.995461 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37451
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:52.322506 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:52.322831 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:52.323009 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:52.330428 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:52.330585 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:52.330771 11775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:41:52.340680 11775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:52.342068 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:52.343436 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:52.357695 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:52.357717 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:52.357985 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:52.358505 11775 server_base.cc:1034] running on GCE node
I20260927 22:41:52.359170 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:52.360292 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:52.361541 11775 hybrid_clock.cc:648] HybridClock initialized: now 1790548912361486 us; error 43 us; skew 500 ppm
I20260927 22:41:52.361838 11775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:52 dist-test-slave-pp80 krb5kdc[11535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:52.365437 11775 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:41:52.367105 11775 webserver.cc:460] Webserver started at http://127.1.53.66:43435/ using document root <none> and password file <none>
I20260927 22:41:52.367825 11775 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:52.367973 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:52.368338 11775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:52.371286 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "91300a05b69a4b1da6ba4d51dfb130c5"
format_stamp: "Formatted at 2026-09-27 22:41:52 on dist-test-slave-pp80"
server_key: "4d045a0398e0f5536df4f74c8d82e72f"
server_key_iv: "32a2394131521f162c9278e57dc225e3"
server_key_version: "encryptionkey@0"
I20260927 22:41:52.372074 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "91300a05b69a4b1da6ba4d51dfb130c5"
format_stamp: "Formatted at 2026-09-27 22:41:52 on dist-test-slave-pp80"
server_key: "4d045a0398e0f5536df4f74c8d82e72f"
server_key_iv: "32a2394131521f162c9278e57dc225e3"
server_key_version: "encryptionkey@0"
I20260927 22:41:52.377727 11775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:41:52.381402 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:52.382782 11775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:41:52.382951 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "91300a05b69a4b1da6ba4d51dfb130c5"
format_stamp: "Formatted at 2026-09-27 22:41:52 on dist-test-slave-pp80"
server_key: "4d045a0398e0f5536df4f74c8d82e72f"
server_key_iv: "32a2394131521f162c9278e57dc225e3"
server_key_version: "encryptionkey@0"
I20260927 22:41:52.383169 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:52.416472 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:52.420919 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:52.421227 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:52.422274 11775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:52.423671 11775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:52.423774 11775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:52.423928 11775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:52.424000 11775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:52.458236 11775 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:34451
I20260927 22:41:52.458287 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:34451 every 8 connection(s)
I20260927 22:41:52.459733 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:41:52.468212 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11775
I20260927 22:41:52.468446 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:41:52.469144 1237 external_mini_cluster.cc:1428] Setting key 672e7029b2cadf7947dedd66a7a8cd05
Sep 27 22:41:52 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:52.481734 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:52.462504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:56539 (local address 127.1.53.126:37451)
0927 22:41:52.462946 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:52.462952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:52.464048 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 22:41:52.466045 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:52.466054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:52.466131 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:52.466233 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:52.467746 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.468584 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.470716 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.471381 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.471501 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:52.475744 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:52.475750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:52.475753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:52.475785 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:52.478512 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:52.479298 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:52.479306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:52.479311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:52.479448 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:52.479890 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:52.479896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:52.479899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:52.480227 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:52.480443 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:52.481140 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 22:41:52.481353 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":208,"threads_started":1}
I20260927 22:41:52.482769 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:52.463208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37451 (local address 127.1.53.66:56539)
0927 22:41:52.463887 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:52.463929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:52.465721 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:52.466464 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:52.466476 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:52.467000 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:52.467579 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:52.467595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.468768 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.468774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:52.470561 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:52.470573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.471640 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.471649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:52.471946 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:52.472685 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:52.472721 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:52.475483 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:52.478668 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:52.478678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:52.478697 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:52.479170 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:52.479576 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:52.479581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:52.479584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:52.479777 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:52.480407 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:52.480415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:52.480771 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 22:41:52.481059 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 22:41:52.481616 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":283,"threads_started":1}
I20260927 22:41:52.484361 11903 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37451
I20260927 22:41:52.484701 11903 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:52.485605 11903 heartbeater.cc:511] Master 127.1.53.126:37451 requested a full tablet report, sending...
I20260927 22:41:52.487349 11580 ts_manager.cc:194] Registered new tserver with Master: 91300a05b69a4b1da6ba4d51dfb130c5 (127.1.53.66:34451)
I20260927 22:41:52.488686 11580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:56539
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:41:52.524128 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37451
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:41:52.857563 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:41:52.857841 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:41:52.858004 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:41:52.865682 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:41:52.865790 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:41:52.866025 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:41:52.876149 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:41:52.877506 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:41:52.878927 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:41:52.892951 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:52.893077 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:41:52.893028 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:41:52.893857 11909 server_base.cc:1034] running on GCE node
I20260927 22:41:52.894484 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:41:52.895462 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:41:52.896706 11909 hybrid_clock.cc:648] HybridClock initialized: now 1790548912896647 us; error 54 us; skew 500 ppm
I20260927 22:41:52.897053 11909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:41:52 dist-test-slave-pp80 krb5kdc[11535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:41:52.900406 11909 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:41:52.901847 11909 webserver.cc:460] Webserver started at http://127.1.53.67:39247/ using document root <none> and password file <none>
I20260927 22:41:52.902559 11909 fs_manager.cc:359] Metadata directory not provided
I20260927 22:41:52.902711 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:41:52.903029 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:41:52.906164 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "f410de82a5ff4c1892da37adb8b2e1d1"
format_stamp: "Formatted at 2026-09-27 22:41:52 on dist-test-slave-pp80"
server_key: "a829bd839753b7291fc14f810e38650f"
server_key_iv: "ed830456636474aa564cbd82354e1aca"
server_key_version: "encryptionkey@0"
I20260927 22:41:52.906999 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "f410de82a5ff4c1892da37adb8b2e1d1"
format_stamp: "Formatted at 2026-09-27 22:41:52 on dist-test-slave-pp80"
server_key: "a829bd839753b7291fc14f810e38650f"
server_key_iv: "ed830456636474aa564cbd82354e1aca"
server_key_version: "encryptionkey@0"
I20260927 22:41:52.912460 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:41:52.916114 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:52.917608 11909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:41:52.917786 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "f410de82a5ff4c1892da37adb8b2e1d1"
format_stamp: "Formatted at 2026-09-27 22:41:52 on dist-test-slave-pp80"
server_key: "a829bd839753b7291fc14f810e38650f"
server_key_iv: "ed830456636474aa564cbd82354e1aca"
server_key_version: "encryptionkey@0"
I20260927 22:41:52.918008 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:41:52.930106 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:41:52.934530 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:41:52.934823 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:41:52.935794 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:41:52.937304 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:41:52.937410 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:52.937573 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:41:52.937651 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:41:52.969278 11769 heartbeater.cc:503] Master 127.1.53.126:37451 was elected leader, sending a full tablet report...
I20260927 22:41:52.972838 11909 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:35007
I20260927 22:41:52.972879 12036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:35007 every 8 connection(s)
I20260927 22:41:52.974179 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:41:52.976635 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 11909
I20260927 22:41:52.976867 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:41:52.977303 1237 external_mini_cluster.cc:1428] Setting key 820397a9bd799d0335eb65ab24124f25
Sep 27 22:41:52 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548912, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:41:52.995297 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:52.976725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:49285 (local address 127.1.53.126:37451)
0927 22:41:52.978029 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:52.978036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:52.978402 (+ 366us) server_negotiation.cc:407] Connection header received
0927 22:41:52.980905 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:52.980911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:52.981001 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:52.981128 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:52.982660 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.983493 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.985514 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.985974 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.986118 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:52.989679 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:52.989683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:52.989687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:52.989716 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:52.992581 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:52.993322 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:52.993325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:52.993328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:52.993439 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:52.993804 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:52.993807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:52.993809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:52.994057 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:52.994163 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:52.994920 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 22:41:52.995083 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1164}
I20260927 22:41:52.996444 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:52.977259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37451 (local address 127.1.53.67:49285)
0927 22:41:52.978250 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:52.978305 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:52.980617 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:52.981376 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:52.981384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:52.981897 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:41:52.982494 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:52.982512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.983938 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.983942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:52.985352 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:52.985369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:52.986222 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:52.986230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:52.986471 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:52.987232 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:41:52.987255 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:41:52.989495 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:41:52.992726 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:52.992737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:52.992753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:52.993208 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:52.993533 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:41:52.993537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:41:52.993539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:41:52.993708 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:41:52.994183 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:41:52.994191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:41:52.994595 (+ 404us) client_negotiation.cc:769] Sending connection context
0927 22:41:52.994828 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 22:41:52.995238 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":376,"threads_started":1}
I20260927 22:41:52.997910 12037 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37451
I20260927 22:41:52.998293 12037 heartbeater.cc:464] Registering TS with master...
I20260927 22:41:52.999033 12037 heartbeater.cc:511] Master 127.1.53.126:37451 requested a full tablet report, sending...
I20260927 22:41:53.000521 11580 ts_manager.cc:194] Registered new tserver with Master: f410de82a5ff4c1892da37adb8b2e1d1 (127.1.53.67:35007)
I20260927 22:41:53.001371 11580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:49285
I20260927 22:41:53.005048 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:41:53.021826 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.009372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.1.53.126:37451)
0927 22:41:53.009541 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.009545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.010030 (+ 485us) server_negotiation.cc:407] Connection header received
0927 22:41:53.010160 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.010163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.010218 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.010297 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:53.011667 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.012464 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.013731 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.014105 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.014190 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.016911 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:53.016915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:53.016919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:53.016947 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:53.019314 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.019973 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.019976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.019978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.020105 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.020435 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.020438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.020440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.020720 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:53.020905 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.021292 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.021509 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:41:53.025946 11580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:41:53.032603 11580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:41:53.058305 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.044398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:34451 (local address 127.0.0.1:39806)
0927 22:41:53.045377 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:53.045423 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:53.045575 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:53.047141 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:53.047153 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:53.047194 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:53.047704 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.047719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.050402 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.050411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.053782 (+ 3371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.053804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.056911 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.056921 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.057235 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.057308 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 22:41:53.057484 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 22:41:53.057762 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":641,"threads_started":1}
I20260927 22:41:53.059690 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.044695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.1.53.66:34451)
0927 22:41:53.046091 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.046104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.046140 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:41:53.046266 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.046278 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.046601 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.046832 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:53.047927 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.050195 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.054022 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.056261 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.056426 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.056717 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.058979 (+ 2262us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.059218 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":1031,"threads_started":1}
I20260927 22:41:53.065135 11838 tablet_service.cc:1467] Processing CreateTablet for tablet b756fe1cf23f4058a15464ef015d5edf (DEFAULT_TABLE table=test-table [id=0833b872e0324769a938e7003d17f916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:53.067186 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b756fe1cf23f4058a15464ef015d5edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:53.078974 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.044793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:35007 (local address 127.0.0.1:58446)
0927 22:41:53.070772 (+ 25979us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:53.070793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:53.070996 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:53.072066 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:53.072071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:53.072096 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:53.072500 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.072513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.074458 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.074464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.076767 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.076781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.078073 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.078079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.078243 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.078277 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:41:53.078398 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 22:41:53.078525 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25874,"mutex_wait_us":28,"thread_start_us":26203,"threads_started":1}
I20260927 22:41:53.079741 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.045956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45595 (local address 127.0.0.1:58010)
0927 22:41:53.071600 (+ 25644us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:53.071625 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:53.071736 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:53.072627 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:53.072631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:53.072655 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:53.073029 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.073040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.074882 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.074887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.076830 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.076842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.078914 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.078921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.079121 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.079155 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:41:53.079243 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:41:53.079369 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25520,"mutex_wait_us":50,"thread_start_us":25380,"threads_started":1}
I20260927 22:41:53.080161 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.046530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.1.53.65:45595)
0927 22:41:53.070311 (+ 23781us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.070319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.071862 (+ 1543us) server_negotiation.cc:407] Connection header received
0927 22:41:53.071969 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.071976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.072223 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.072642 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:53.073231 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.074710 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.076986 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.078425 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.078551 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.078757 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.079639 (+ 882us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.079795 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23648,"thread_start_us":23766,"threads_started":1}
I20260927 22:41:53.080216 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.045486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.1.53.67:35007)
0927 22:41:53.069934 (+ 24448us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.069941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.070941 (+ 1000us) server_negotiation.cc:407] Connection header received
0927 22:41:53.071215 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.071224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.071509 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.071729 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:53.072726 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.074299 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.076940 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.077868 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.078021 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.078257 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.079530 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.079714 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24273,"thread_start_us":24121,"threads_started":1}
I20260927 22:41:53.083894 12055 tablet_bootstrap.cc:492] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Bootstrap starting.
I20260927 22:41:53.083930 11972 tablet_service.cc:1467] Processing CreateTablet for tablet b756fe1cf23f4058a15464ef015d5edf (DEFAULT_TABLE table=test-table [id=0833b872e0324769a938e7003d17f916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:53.083927 11704 tablet_service.cc:1467] Processing CreateTablet for tablet b756fe1cf23f4058a15464ef015d5edf (DEFAULT_TABLE table=test-table [id=0833b872e0324769a938e7003d17f916]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:41:53.085269 11972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b756fe1cf23f4058a15464ef015d5edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:53.085381 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b756fe1cf23f4058a15464ef015d5edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:41:53.087641 12055 tablet_bootstrap.cc:654] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:53.088950 12055 log.cc:824] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:53.092074 12055 tablet_bootstrap.cc:492] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: No bootstrap required, opened a new log
I20260927 22:41:53.092370 12055 ts_tablet_manager.cc:1397] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:41:53.097242 12057 tablet_bootstrap.cc:492] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Bootstrap starting.
I20260927 22:41:53.099489 12058 tablet_bootstrap.cc:492] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Bootstrap starting.
I20260927 22:41:53.100739 12057 tablet_bootstrap.cc:654] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:53.102329 12055 raft_consensus.cc:348] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.102648 12057 log.cc:824] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:53.102710 12055 raft_consensus.cc:374] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:53.102854 12055 raft_consensus.cc:729] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91300a05b69a4b1da6ba4d51dfb130c5, State: Initialized, Role: FOLLOWER
I20260927 22:41:53.103055 12058 tablet_bootstrap.cc:654] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Neither blocks nor log segments found. Creating new log.
I20260927 22:41:53.103468 12055 consensus_queue.cc:260] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.104629 11903 heartbeater.cc:503] Master 127.1.53.126:37451 was elected leader, sending a full tablet report...
I20260927 22:41:53.105073 12055 ts_tablet_manager.cc:1428] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:41:53.105448 12058 log.cc:824] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:41:53.108338 12058 tablet_bootstrap.cc:492] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: No bootstrap required, opened a new log
I20260927 22:41:53.108592 12058 ts_tablet_manager.cc:1397] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 22:41:53.111017 12057 tablet_bootstrap.cc:492] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: No bootstrap required, opened a new log
I20260927 22:41:53.111285 12057 ts_tablet_manager.cc:1397] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260927 22:41:53.119180 12058 raft_consensus.cc:348] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.119635 12058 raft_consensus.cc:374] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:53.119736 12058 raft_consensus.cc:729] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928c08f00de04dfcbbf7cdd0ae667a1b, State: Initialized, Role: FOLLOWER
I20260927 22:41:53.120441 12058 consensus_queue.cc:260] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.121635 12057 raft_consensus.cc:348] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.122081 12057 raft_consensus.cc:374] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:41:53.122184 12057 raft_consensus.cc:729] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f410de82a5ff4c1892da37adb8b2e1d1, State: Initialized, Role: FOLLOWER
I20260927 22:41:53.122483 12058 ts_tablet_manager.cc:1428] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 22:41:53.122776 12057 consensus_queue.cc:260] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.123926 12037 heartbeater.cc:503] Master 127.1.53.126:37451 was elected leader, sending a full tablet report...
I20260927 22:41:53.124470 12057 ts_tablet_manager.cc:1428] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 22:41:53.154525 12062 raft_consensus.cc:482] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:41:53.154820 12062 raft_consensus.cc:504] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.156710 12062 leader_election.cc:288] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91300a05b69a4b1da6ba4d51dfb130c5 (127.1.53.66:34451), f410de82a5ff4c1892da37adb8b2e1d1 (127.1.53.67:35007)
I20260927 22:41:53.163321 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.157154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:38881 (local address 127.1.53.66:34451)
0927 22:41:53.157517 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.157524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.157845 (+ 321us) server_negotiation.cc:407] Connection header received
0927 22:41:53.158084 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.158088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.158143 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.158238 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:53.158838 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.159650 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.161643 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.162413 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.162570 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.162665 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.162940 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.163061 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260927 22:41:53.163388 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.156918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:34451 (local address 127.1.53.65:38881)
0927 22:41:53.157734 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:53.157750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:53.157939 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:53.158264 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:53.158269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:53.158293 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:53.158689 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.158712 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.159775 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.159779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.161496 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.161507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.162624 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.162629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.162783 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.162805 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:41:53.162917 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:41:53.163033 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":489,"threads_started":1}
I20260927 22:41:53.163404 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.157112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:33355 (local address 127.1.53.67:35007)
0927 22:41:53.157317 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.157326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.158133 (+ 807us) server_negotiation.cc:407] Connection header received
0927 22:41:53.158328 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.158332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.158388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.158485 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:41:53.159083 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.159809 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.161770 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.162506 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.162654 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.162763 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.163031 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.163157 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:41:53.163618 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.157106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:35007 (local address 127.1.53.65:33355)
0927 22:41:53.158016 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 22:41:53.158040 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:41:53.158184 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:41:53.158500 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:41:53.158504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:41:53.158526 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:41:53.158905 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.158913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.159950 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.159956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.161617 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:41:53.161629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.162739 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.162747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:41:53.162887 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.162907 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:41:53.163008 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:41:53.163129 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":611,"threads_started":1}
I20260927 22:41:53.164590 11992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b756fe1cf23f4058a15464ef015d5edf" candidate_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" is_pre_election: true
I20260927 22:41:53.164623 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b756fe1cf23f4058a15464ef015d5edf" candidate_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" is_pre_election: true
I20260927 22:41:53.165133 11992 raft_consensus.cc:2454] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928c08f00de04dfcbbf7cdd0ae667a1b in term 0.
I20260927 22:41:53.165135 11858 raft_consensus.cc:2454] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928c08f00de04dfcbbf7cdd0ae667a1b in term 0.
I20260927 22:41:53.165755 11660 leader_election.cc:302] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91300a05b69a4b1da6ba4d51dfb130c5, 928c08f00de04dfcbbf7cdd0ae667a1b; no voters:
I20260927 22:41:53.166180 12062 raft_consensus.cc:2781] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:41:53.166352 12062 raft_consensus.cc:482] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:41:53.166481 12062 raft_consensus.cc:3037] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:53.168604 12062 raft_consensus.cc:504] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.169222 12062 leader_election.cc:288] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [CANDIDATE]: Term 1 election: Requested vote from peers 91300a05b69a4b1da6ba4d51dfb130c5 (127.1.53.66:34451), f410de82a5ff4c1892da37adb8b2e1d1 (127.1.53.67:35007)
I20260927 22:41:53.169739 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b756fe1cf23f4058a15464ef015d5edf" candidate_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91300a05b69a4b1da6ba4d51dfb130c5"
I20260927 22:41:53.169782 11992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b756fe1cf23f4058a15464ef015d5edf" candidate_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f410de82a5ff4c1892da37adb8b2e1d1"
I20260927 22:41:53.169953 11858 raft_consensus.cc:3037] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:53.169953 11992 raft_consensus.cc:3037] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:41:53.173213 11992 raft_consensus.cc:2454] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928c08f00de04dfcbbf7cdd0ae667a1b in term 1.
I20260927 22:41:53.173218 11858 raft_consensus.cc:2454] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928c08f00de04dfcbbf7cdd0ae667a1b in term 1.
I20260927 22:41:53.173601 11660 leader_election.cc:302] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91300a05b69a4b1da6ba4d51dfb130c5, 928c08f00de04dfcbbf7cdd0ae667a1b; no voters:
I20260927 22:41:53.173974 12062 raft_consensus.cc:2781] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:41:53.174441 12062 raft_consensus.cc:686] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 1 LEADER]: Becoming Leader. State: Replica: 928c08f00de04dfcbbf7cdd0ae667a1b, State: Running, Role: LEADER
I20260927 22:41:53.174983 12062 consensus_queue.cc:237] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } }
I20260927 22:41:53.179360 11579 catalog_manager.cc:5495] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 928c08f00de04dfcbbf7cdd0ae667a1b (127.1.53.65). New cstate: current_term: 1 leader_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "928c08f00de04dfcbbf7cdd0ae667a1b" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45595 } health_report { overall_health: HEALTHY } } }
W20260927 22:41:53.187245 11770 tablet.cc:2369] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:53.206418 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.201116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.1.53.65:45595)
0927 22:41:53.201337 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.201352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.201454 (+ 102us) server_negotiation.cc:407] Connection header received
0927 22:41:53.201704 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.201708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.201783 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.201935 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:41:53.202283 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.202974 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.204349 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.204748 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.204880 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.205348 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:41:53.205851 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:41:53.206010 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.206068 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.206147 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:41:53.212060 11904 tablet.cc:2369] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:53.214720 11992 raft_consensus.cc:1261] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 1 FOLLOWER]: Refusing update from remote peer 928c08f00de04dfcbbf7cdd0ae667a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:53.214720 11858 raft_consensus.cc:1261] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 1 FOLLOWER]: Refusing update from remote peer 928c08f00de04dfcbbf7cdd0ae667a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:41:53.215838 12062 consensus_queue.cc:1035] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f410de82a5ff4c1892da37adb8b2e1d1" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:41:53.216192 12066 consensus_queue.cc:1035] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "91300a05b69a4b1da6ba4d51dfb130c5" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:41:53.225952 12038 tablet.cc:2369] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:41:53.227367 12069 mvcc.cc:205] Tried to move back new op lower bound from 7334088348516868096 to 7334088348368695296. Current Snapshot: MvccSnapshot[applied={T|T < 7334088348516868096}]
I20260927 22:41:53.227783 12071 mvcc.cc:205] Tried to move back new op lower bound from 7334088348516868096 to 7334088348368695296. Current Snapshot: MvccSnapshot[applied={T|T < 7334088348516868096}]
I20260927 22:41:53.230703 12070 mvcc.cc:205] Tried to move back new op lower bound from 7334088348516868096 to 7334088348368695296. Current Snapshot: MvccSnapshot[applied={T|T < 7334088348516868096}]
I20260927 22:41:53.241822 11580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:53.242132 11580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:41:53.245746 11580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d19978d4b58245749c55cfb06038e1f8: Sending DeleteTablet for 3 replicas of tablet b756fe1cf23f4058a15464ef015d5edf
I20260927 22:41:53.247124 11972 tablet_service.cc:1514] Processing DeleteTablet for tablet b756fe1cf23f4058a15464ef015d5edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:53 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:58446
I20260927 22:41:53.247124 11838 tablet_service.cc:1514] Processing DeleteTablet for tablet b756fe1cf23f4058a15464ef015d5edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:53 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:39806
I20260927 22:41:53.247511 11704 tablet_service.cc:1514] Processing DeleteTablet for tablet b756fe1cf23f4058a15464ef015d5edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:41:53 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:58010
I20260927 22:41:53.249461 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11641
I20260927 22:41:53.258731 12079 tablet_replica.cc:330] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: stopping tablet replica
I20260927 22:41:53.259488 12079 raft_consensus.cc:2229] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:53.260347 12079 raft_consensus.cc:2258] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:53.261560 12080 tablet_replica.cc:330] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: stopping tablet replica
I20260927 22:41:53.261773 12081 tablet_replica.cc:330] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: stopping tablet replica
I20260927 22:41:53.262095 12081 raft_consensus.cc:2229] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:41:53.262079 12080 raft_consensus.cc:2229] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:41:53.262602 12080 raft_consensus.cc:2258] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:53.262763 12081 raft_consensus.cc:2258] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:41:53.263345 12079 ts_tablet_manager.cc:1905] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:53.265223 12081 ts_tablet_manager.cc:1905] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:53.266570 12080 ts_tablet_manager.cc:1905] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:41:53.270778 12079 ts_tablet_manager.cc:1918] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:53.270948 12079 log.cc:1196] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/b756fe1cf23f4058a15464ef015d5edf
I20260927 22:41:53.271514 12079 ts_tablet_manager.cc:1939] T b756fe1cf23f4058a15464ef015d5edf P 91300a05b69a4b1da6ba4d51dfb130c5: Deleting consensus metadata
I20260927 22:41:53.272528 12081 ts_tablet_manager.cc:1918] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:53.272697 12081 log.cc:1196] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/b756fe1cf23f4058a15464ef015d5edf
I20260927 22:41:53.273187 12081 ts_tablet_manager.cc:1939] T b756fe1cf23f4058a15464ef015d5edf P 928c08f00de04dfcbbf7cdd0ae667a1b: Deleting consensus metadata
I20260927 22:41:53.273648 11567 catalog_manager.cc:4841] TS 91300a05b69a4b1da6ba4d51dfb130c5 (127.1.53.66:34451): tablet b756fe1cf23f4058a15464ef015d5edf (table test-table [id=0833b872e0324769a938e7003d17f916]) successfully deleted
I20260927 22:41:53.274777 11564 catalog_manager.cc:4841] TS 928c08f00de04dfcbbf7cdd0ae667a1b (127.1.53.65:45595): tablet b756fe1cf23f4058a15464ef015d5edf (table test-table [id=0833b872e0324769a938e7003d17f916]) successfully deleted
Sep 27 22:41:53 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:41:53.276005 12080 ts_tablet_manager.cc:1918] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:41:53.276150 12080 log.cc:1196] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/b756fe1cf23f4058a15464ef015d5edf
I20260927 22:41:53.276683 12080 ts_tablet_manager.cc:1939] T b756fe1cf23f4058a15464ef015d5edf P f410de82a5ff4c1892da37adb8b2e1d1: Deleting consensus metadata
I20260927 22:41:53.278245 11565 catalog_manager.cc:4841] TS f410de82a5ff4c1892da37adb8b2e1d1 (127.1.53.67:35007): tablet b756fe1cf23f4058a15464ef015d5edf (table test-table [id=0833b872e0324769a938e7003d17f916]) successfully deleted
I20260927 22:41:53.282238 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.250287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.1.53.65:45595)
0927 22:41:53.250464 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.250472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.264494 (+ 14022us) server_negotiation.cc:407] Connection header received
0927 22:41:53.264782 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.264788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.264898 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.264988 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:53.267103 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.268287 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.270250 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.271033 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.271157 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.276139 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:53.276158 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:53.276176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:53.276217 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:53.279414 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.280074 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.280089 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.280092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.280308 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.280694 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.280701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.280705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.281045 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:53.281349 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.281630 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.281860 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:41:53.282838 11764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:53.380906 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11641
I20260927 22:41:53.399673 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11775
Sep 27 22:41:53 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:41:53.414502 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.400548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.1.53.66:34451)
0927 22:41:53.400866 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.400874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.400900 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:41:53.401066 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.401074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.401183 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.401309 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:53.402501 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.403275 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.404589 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.404959 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.405050 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.408996 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:53.409008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:53.409023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:53.409053 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:53.411876 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.412704 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.412712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.412714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.412838 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.413263 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.413275 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.413278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.413636 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:53.413849 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.414090 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.414248 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260927 22:41:53.415277 11898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:53.504036 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11775
I20260927 22:41:53.521312 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11909
Sep 27 22:41:53 dist-test-slave-pp80 krb5kdc[11535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:41:53.535755 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:41:53.522022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.1.53.67:35007)
0927 22:41:53.522377 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 22:41:53.522382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:41:53.522422 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:41:53.522477 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:41:53.522481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:41:53.522550 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:41:53.522633 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:41:53.524045 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.524837 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.526079 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:41:53.526461 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:41:53.526561 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:41:53.530656 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:41:53.530671 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:41:53.530686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:41:53.530717 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:41:53.533455 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.534063 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.534083 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.534085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.534206 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:41:53.534541 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:41:53.534544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:41:53.534546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:41:53.534847 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:41:53.535049 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:41:53.535333 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:41:53.535490 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260927 22:41:53.536499 12032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:53.624975 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11909
I20260927 22:41:53.643290 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 11548
I20260927 22:41:53.644506 11610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:41:53.755251 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 11548
2026-09-27T22:41:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4232 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:41:53 dist-test-slave-pp80 krb5kdc[12089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:41:53 dist-test-slave-pp80 krb5kdc[12089](info): set up 2 sockets
Sep 27 22:41:53 dist-test-slave-pp80 krb5kdc[12089](info): commencing operation
krb5kdc: starting...
W20260927 22:41:54.869943 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:41:55 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:41:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/rangeradmin_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/rangeradmin_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/rangerlookup_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/rangerlookup_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
I20260927 22:41:55.209044 1237 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkEfv30/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/postgres -l logfile start
2026-09-27 22:41:57.922 UTC [12119] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:41:57.922 UTC [12119] LOG: listening on IPv4 address "127.1.53.84", port 51209
2026-09-27 22:41:57.926 UTC [12119] LOG: listening on Unix socket "/tmp/.s.PGSQL.51209"
2026-09-27 22:41:57.948 UTC [12122] LOG: database system was shut down at 2026-09-27 22:41:56 UTC
2026-09-27 22:41:57.951 UTC [12119] LOG: database system is ready to accept connections
I20260927 22:41:58.924973 1237 mini_postgres.cc:96] Postgres bound to 51209
2026-09-27 22:41:58.943 UTC [12130] FATAL: database "slave" does not exist
127.1.53.84:51209 - accepting connections
I20260927 22:41:58.944118 1237 mini_ranger.cc:161] Starting Ranger...
I20260927 22:41:58.973912 1237 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:41:59.528501 1237 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:41:59.528612 1237 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-admin
2026-09-27 22:41:59,823 [I] DB FLAVOR :POSTGRES
2026-09-27 22:41:59,823 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:41:59,823 [I] Checking connection..
2026-09-27 22:41:59,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:42:00,071 [I] Checking connection passed.
2026-09-27 22:42:00,072 [I] --------- Verifying version history table ---------
2026-09-27 22:42:00,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:42:00,340 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:42:00,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:42:00,630 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:42:00,631 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:42:00,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:42:00.880 UTC [12194] WARNING: there is no transaction in progress
2026-09-27 22:42:00,886 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:42:00,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:42:01,145 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:42:01,145 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:42:01,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:42:01,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:42:01,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pp80.c.gcp-upstream.internal','N') ;"
2026-09-27 22:42:01,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:42:02,159 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:42:02,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:42:02,416 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:42:02,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:42:02,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:42:02,921 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:42:02,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:42:03.922 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:03.939 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:03.956 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:03.971 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:03.989 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.010 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.071 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.079 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.094 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.104 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.115 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.131 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.142 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.149 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.156 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04.163 UTC [12302] WARNING: there is no transaction in progress
2026-09-27 22:42:04,835 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:42:04,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:42:05,064 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:42:05,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:42:05,318 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:42:05,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:42:05,582 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:42:05,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:42:05,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:42:06,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pp80.c.gcp-upstream.internal';"
2026-09-27 22:42:06,310 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:42:06,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:42:06,559 [I] DB_PATCHES have already been applied
I20260927 22:42:06.565671 1237 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkEfv30/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:42:06.565755 1237 mini_ranger.cc:193] Using host: 127.1.53.84
I20260927 22:42:06.580755 1237 mini_ranger.cc:227] Ranger admin URL: http://127.1.53.84:42217
Sep 27, 2026 10:42:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:42:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:42:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:42:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:42:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:42:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:42:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:42:07.734897 1237 mini_ranger.cc:160] Time spent starting Ranger: real 8.791s user 0.005s sys 0.034s
Sep 27, 2026 10:42:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:42:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:42:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:42:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:42:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:42:29.935719 1237 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/rangerkms_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/rangerkms_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.53.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.53.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:42:30 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548950, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:42:30.114962 1237 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:42:30.137076 1237 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:42:31.124963 1237 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:42:31.125139 1237 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms
I20260927 22:42:31.125248 1237 mini_ranger_kms.cc:227] Using postgres at 127.1.53.84:51209
2026-09-27 22:42:31,537 [I] DB FLAVOR :POSTGRES
2026-09-27 22:42:31,538 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:42:31,538 [I] Checking connection
2026-09-27 22:42:31,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:42:31,786 [I] connection success
2026-09-27 22:42:31,786 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:42:31,786 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:42:31,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:42:32,028 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:42:32,028 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:42:32,028 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:42:32,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:51209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:42:32,280 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:42:32.767081 1237 mini_ranger_kms.cc:310] Created kms service
I20260927 22:42:32.956091 1237 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:42:33.002113 1237 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:42:33 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548950, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:42:33.404407 1237 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:42:33.404763 1237 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:42:33.404831 1237 mini_ranger_kms.cc:240] Using host: 127.1.53.84
I20260927 22:42:33.426899 1237 mini_ranger_kms.cc:276] Ranger KMS PID: 12483
I20260927 22:42:33.427095 1237 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.53.84:43331
Sep 27, 2026 10:42:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:42:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:42:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:42:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:42:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:42:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:42:35.747228 1237 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.632s user 0.086s sys 0.343s
I20260927 22:42:35.747504 1237 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:42:35.747732 1237 mini_ranger_kms.cc:401] 127.1.53.84:43331/kms/v1/keys
Sep 27 22:42:35 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790548955, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.53.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:42:36 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548955, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.53.84@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
Sep 27 22:42:36 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548955, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.53.84@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
Sep 27 22:42:39 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:42:39.184648 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37647
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37647
--ranger_config_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:43331/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:42:39.527185 12533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:42:39.527546 12533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:42:39.527724 12533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:42:39.535521 12533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:42:39.535701 12533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:42:39.535806 12533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:42:39.535910 12533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:42:39.536002 12533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:42:39.546232 12533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.53.84:43331/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37647
--ranger_config_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:42:39.547632 12533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:42:39.549015 12533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:42:39.561121 12538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:39.561174 12539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:39.561200 12541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:42:39.562291 12533 server_base.cc:1034] running on GCE node
I20260927 22:42:39.563100 12533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:42:39.564656 12533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:42:39.565888 12533 hybrid_clock.cc:648] HybridClock initialized: now 1790548959565826 us; error 59 us; skew 500 ppm
I20260927 22:42:39.566233 12533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:42:39 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:42:39.575181 12533 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:42:39.576915 12533 webserver.cc:460] Webserver started at http://127.1.53.126:45017/ using document root <none> and password file <none>
I20260927 22:42:39.577633 12533 fs_manager.cc:359] Metadata directory not provided
I20260927 22:42:39.577795 12533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:42:39.578136 12533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:42:39 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:42:39.645219 12533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "515d8ae583f14b2eac78dbb250130999"
format_stamp: "Formatted at 2026-09-27 22:42:39 on dist-test-slave-pp80"
server_key: "a2e49f5a7a9b324cc30a9659cbc7f281"
server_key_iv: "c50d0c00a0b59913d3a1b69fe4493d15"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:39.646076 12533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "515d8ae583f14b2eac78dbb250130999"
format_stamp: "Formatted at 2026-09-27 22:42:39 on dist-test-slave-pp80"
server_key: "a2e49f5a7a9b324cc30a9659cbc7f281"
server_key_iv: "c50d0c00a0b59913d3a1b69fe4493d15"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:39.651515 12533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:42:39.671795 12548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:39.673466 12533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:42:39.673650 12533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "515d8ae583f14b2eac78dbb250130999"
format_stamp: "Formatted at 2026-09-27 22:42:39 on dist-test-slave-pp80"
server_key: "a2e49f5a7a9b324cc30a9659cbc7f281"
server_key_iv: "c50d0c00a0b59913d3a1b69fe4493d15"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:39.673871 12533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:42:39.702279 12533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:42:39.706598 12533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:42:39.706920 12533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:42:39.728168 12533 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37647
I20260927 22:42:39.728189 12599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37647 every 8 connection(s)
I20260927 22:42:39.729568 12533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:42:39.734033 12600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:42:39.734839 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 12533
I20260927 22:42:39.735040 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:42:39.735853 1237 external_mini_cluster.cc:1428] Setting key 88ceb57050b11866e920bc73e1edd8ab
I20260927 22:42:39.753075 12600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999: Bootstrap starting.
I20260927 22:42:39.757123 12600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999: Neither blocks nor log segments found. Creating new log.
I20260927 22:42:39.758276 12600 log.cc:824] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999: Log is configured to *not* fsync() on all Append() calls
I20260927 22:42:39.761735 12600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999: No bootstrap required, opened a new log
Sep 27 22:42:39 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:42:39.772444 12600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d8ae583f14b2eac78dbb250130999" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37647 } }
I20260927 22:42:39.772848 12600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:42:39.772969 12600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515d8ae583f14b2eac78dbb250130999, State: Initialized, Role: FOLLOWER
I20260927 22:42:39.773639 12600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d8ae583f14b2eac78dbb250130999" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37647 } }
I20260927 22:42:39.773895 12600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:42:39.774036 12600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:42:39.774238 12600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:42:39.776862 12600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d8ae583f14b2eac78dbb250130999" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37647 } }
I20260927 22:42:39.776969 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:39.738096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.1.53.126:37647)
0927 22:42:39.748922 (+ 10826us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:39.748943 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:39.752227 (+ 3284us) server_negotiation.cc:407] Connection header received
0927 22:42:39.754833 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:39.754865 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:39.755466 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:39.755925 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:39.757443 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:39.759626 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:39.761470 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:39.762472 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:39.762651 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:39.767862 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:39.767877 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:39.767904 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:39.767963 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:39.771927 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:39.772798 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:39.772810 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:39.772820 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:39.773023 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:39.773447 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:39.773452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:39.773455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:39.774147 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:39.774427 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:39.774990 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 22:42:39.775584 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10539,"thread_start_us":10292,"threads_started":1}
I20260927 22:42:39.777488 12600 leader_election.cc:302] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515d8ae583f14b2eac78dbb250130999; no voters:
I20260927 22:42:39.778242 12600 leader_election.cc:288] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:42:39.778419 12605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:42:39.780975 12605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [term 1 LEADER]: Becoming Leader. State: Replica: 515d8ae583f14b2eac78dbb250130999, State: Running, Role: LEADER
I20260927 22:42:39.781762 12605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d8ae583f14b2eac78dbb250130999" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37647 } }
I20260927 22:42:39.783706 12600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:42:39.784834 12606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515d8ae583f14b2eac78dbb250130999. Latest consensus state: current_term: 1 leader_uuid: "515d8ae583f14b2eac78dbb250130999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d8ae583f14b2eac78dbb250130999" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37647 } } }
I20260927 22:42:39.785414 12606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [sys.catalog]: This master's current role is: LEADER
I20260927 22:42:39.785058 12607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515d8ae583f14b2eac78dbb250130999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d8ae583f14b2eac78dbb250130999" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37647 } } }
I20260927 22:42:39.785805 12607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999 [sys.catalog]: This master's current role is: LEADER
I20260927 22:42:39.792595 12610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:42:39.801164 12610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:42:39.814571 12610 catalog_manager.cc:1347] Generated new cluster ID: 8d5982490e354ece96a4cc9ec226c6b0
I20260927 22:42:39.814703 12610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:42:39.830113 12600 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:42:39.852077 12610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:42:39.854620 12610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:42:39.876066 12610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999: Generated new TSK 0
I20260927 22:42:39.877956 12610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:42:40.681885 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:43331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37647
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:42:40 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790548960, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:42:41.119048 12645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:42:41.119563 12645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:42:41.119783 12645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:42:41.132387 12645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:42:41.132633 12645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:42:41.132899 12645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:42:41.150096 12645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.53.84:43331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:42:41.151866 12645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:42:41.153625 12645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:42:41.173046 12655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:41.173668 12656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:41.179085 12658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:42:41.179679 12645 server_base.cc:1034] running on GCE node
I20260927 22:42:41.180588 12645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:42:41.188587 12645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:42:41.190913 12645 hybrid_clock.cc:648] HybridClock initialized: now 1790548961190824 us; error 79 us; skew 500 ppm
I20260927 22:42:41.191426 12645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:42:41 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548961, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:42:41.196080 12645 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:42:41.199108 12645 webserver.cc:460] Webserver started at http://127.1.53.65:41927/ using document root <none> and password file <none>
I20260927 22:42:41.200141 12645 fs_manager.cc:359] Metadata directory not provided
I20260927 22:42:41.200322 12645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:42:41.200736 12645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:42:41 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548961, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:42:41.225375 12645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "19e43bb1156d474da1a85ed77af08435"
format_stamp: "Formatted at 2026-09-27 22:42:41 on dist-test-slave-pp80"
server_key: "b4c4ef2a498322527588df36367f3682"
server_key_iv: "6cf65ad6bd00f3ec646a95b829b65b3a"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:41.226595 12645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "19e43bb1156d474da1a85ed77af08435"
format_stamp: "Formatted at 2026-09-27 22:42:41 on dist-test-slave-pp80"
server_key: "b4c4ef2a498322527588df36367f3682"
server_key_iv: "6cf65ad6bd00f3ec646a95b829b65b3a"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:41.234699 12645 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260927 22:42:41.252694 12665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:41.254463 12645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:42:41.254678 12645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "19e43bb1156d474da1a85ed77af08435"
format_stamp: "Formatted at 2026-09-27 22:42:41 on dist-test-slave-pp80"
server_key: "b4c4ef2a498322527588df36367f3682"
server_key_iv: "6cf65ad6bd00f3ec646a95b829b65b3a"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:41.254927 12645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:42:41.332345 12645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:42:41.338590 12645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:42:41.338960 12645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:42:41.343732 12645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:42:41.345463 12645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:42:41.345577 12645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:41.345701 12645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:42:41.345788 12645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:41.403043 12645 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:32887
I20260927 22:42:41.403064 12777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:32887 every 8 connection(s)
I20260927 22:42:41.404894 12645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:42:41.412058 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 12645
I20260927 22:42:41.412323 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:42:41.413331 1237 external_mini_cluster.cc:1428] Setting key 9eeec50063a908785fa2f51c1c551ca8
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Sep 27 22:42:41 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548961, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:42:41.479493 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:41.409178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:57025 (local address 127.1.53.126:37647)
0927 22:42:41.448645 (+ 39467us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:41.448653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:41.448680 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:42:41.448754 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:41.448760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:41.448855 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:41.448986 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:41.460119 (+ 11133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:41.461476 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:41.465061 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:41.465697 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:41.466044 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:41.470857 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:41.470864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:41.470869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:41.470924 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:41.474379 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:41.475346 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:41.475351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:41.475355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:41.475974 (+ 619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:41.476537 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:41.476542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:41.476546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:41.476889 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:41.477012 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:41.478878 (+ 1866us) server_negotiation.cc:299] Negotiation successful
0927 22:42:41.479113 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39317,"thread_start_us":39167,"threads_started":1}
I20260927 22:42:41.481040 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:41.409349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37647 (local address 127.1.53.65:57025)
0927 22:42:41.435492 (+ 26143us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:41.435571 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:41.438533 (+ 2962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:41.449579 (+ 11046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:41.449598 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:41.458344 (+ 8746us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:42:41.459612 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:41.459652 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:41.461742 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:41.461748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:41.464376 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:41.464405 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:41.466376 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:41.466383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:41.466712 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:41.467488 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:42:41.467519 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:42:41.470564 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:42:41.474572 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:42:41.474582 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:42:41.474604 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:42:41.475195 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:42:41.476116 (+ 921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:42:41.476121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:42:41.476124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:42:41.476401 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:42:41.477095 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:42:41.477103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:42:41.477606 (+ 503us) client_negotiation.cc:769] Sending connection context
0927 22:42:41.477924 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 22:42:41.478476 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25706,"thread_start_us":29579,"threads_started":1}
Sep 27 22:42:41 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548960, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:42:41.483248 12778 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37647
I20260927 22:42:41.483583 12778 heartbeater.cc:464] Registering TS with master...
I20260927 22:42:41.484512 12778 heartbeater.cc:511] Master 127.1.53.126:37647 requested a full tablet report, sending...
I20260927 22:42:41.487182 12565 ts_manager.cc:194] Registered new tserver with Master: 19e43bb1156d474da1a85ed77af08435 (127.1.53.65:32887)
I20260927 22:42:41.489416 12565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:57025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:42:41.513142 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:43331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37647
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:42:41 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548960, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
W20260927 22:42:41.966640 12784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:42:41.967082 12784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:42:41.967254 12784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:42:41.979131 12784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:42:41.979359 12784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:42:41.979610 12784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:42:41.995353 12784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.53.84:43331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:42:41.997526 12784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:42:41.999395 12784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:42:42.023988 12792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:42.030171 12791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:42.032512 12794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:42:42.034183 12784 server_base.cc:1034] running on GCE node
I20260927 22:42:42.034977 12784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:42:42.036386 12784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:42:42.037719 12784 hybrid_clock.cc:648] HybridClock initialized: now 1790548962037605 us; error 104 us; skew 500 ppm
I20260927 22:42:42.038125 12784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:42:42 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:42:42.042464 12784 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:42:42.044615 12784 webserver.cc:460] Webserver started at http://127.1.53.66:46873/ using document root <none> and password file <none>
I20260927 22:42:42.045529 12784 fs_manager.cc:359] Metadata directory not provided
I20260927 22:42:42.045665 12784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:42:42.046038 12784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:42:42 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:42:42.065174 12784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "dd01b29c2bc2482a87f3b0c154e86f82"
format_stamp: "Formatted at 2026-09-27 22:42:42 on dist-test-slave-pp80"
server_key: "b153f087294fb752be2c3da3a64f299c"
server_key_iv: "8903505f0c61d310ff0a9e58ce4b8cd6"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:42.066344 12784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "dd01b29c2bc2482a87f3b0c154e86f82"
format_stamp: "Formatted at 2026-09-27 22:42:42 on dist-test-slave-pp80"
server_key: "b153f087294fb752be2c3da3a64f299c"
server_key_iv: "8903505f0c61d310ff0a9e58ce4b8cd6"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:42.082489 12784 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.013s sys 0.000s
I20260927 22:42:42.101890 12807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:42.104344 12784 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:42:42.104610 12784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "dd01b29c2bc2482a87f3b0c154e86f82"
format_stamp: "Formatted at 2026-09-27 22:42:42 on dist-test-slave-pp80"
server_key: "b153f087294fb752be2c3da3a64f299c"
server_key_iv: "8903505f0c61d310ff0a9e58ce4b8cd6"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:42.104943 12784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:42:42.126447 12784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:42:42.130972 12784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:42:42.131299 12784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:42:42.132905 12784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:42:42.134496 12784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:42:42.134595 12784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:42.134742 12784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:42:42.134811 12784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:42.174537 12784 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:37831
I20260927 22:42:42.174554 12920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:37831 every 8 connection(s)
I20260927 22:42:42.175909 12784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:42:42.182610 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 12784
I20260927 22:42:42.182783 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:42:42.183293 1237 external_mini_cluster.cc:1428] Setting key 9b79daad03659d78940617898c6503b6
Sep 27 22:42:42 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:42:42.199414 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.179047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:59563 (local address 127.1.53.126:37647)
0927 22:42:42.179537 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:42.179544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:42.180846 (+ 1302us) server_negotiation.cc:407] Connection header received
0927 22:42:42.183153 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:42.183159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:42.183263 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:42.183382 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:42.186110 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.186831 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.188475 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.188859 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.189004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.192336 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:42.192343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:42.192347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:42.192386 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:42.195560 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:42.196566 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:42.196572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:42.196575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:42.196723 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:42.197267 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:42.197272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:42.197275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:42.197599 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:42.197757 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:42.198768 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0927 22:42:42.199029 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260927 22:42:42.200773 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.179740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37647 (local address 127.1.53.66:59563)
0927 22:42:42.180647 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:42.180702 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:42.182847 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:42.183677 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:42.183686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:42.184370 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:42:42.185944 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.185964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.186984 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.186990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.188308 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.188331 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.189084 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.189091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.189345 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.190068 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:42:42.190090 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:42:42.192084 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:42:42.195701 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:42:42.195712 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:42:42.195734 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:42:42.196406 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:42:42.196846 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:42:42.196854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:42:42.196861 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:42:42.197140 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:42:42.197808 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:42:42.197819 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:42:42.198301 (+ 482us) client_negotiation.cc:769] Sending connection context
0927 22:42:42.198620 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:42:42.199218 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":450,"threads_started":1}
I20260927 22:42:42.202394 12921 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37647
I20260927 22:42:42.202737 12921 heartbeater.cc:464] Registering TS with master...
I20260927 22:42:42.203648 12921 heartbeater.cc:511] Master 127.1.53.126:37647 requested a full tablet report, sending...
I20260927 22:42:42.205552 12565 ts_manager.cc:194] Registered new tserver with Master: dd01b29c2bc2482a87f3b0c154e86f82 (127.1.53.66:37831)
I20260927 22:42:42.206691 12565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:59563
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:42:42.244421 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:43331/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37647
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:42:42.493405 12778 heartbeater.cc:503] Master 127.1.53.126:37647 was elected leader, sending a full tablet report...
W20260927 22:42:42.596485 12927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:42:42.596822 12927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:42:42.597000 12927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:42:42.604197 12927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:42:42.604385 12927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:42:42.604579 12927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:42:42.614915 12927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.53.84:43331/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:42:42.616312 12927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:42:42.617666 12927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:42:42.631302 12935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:42.631443 12933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:42:42.631417 12932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:42:42.632719 12927 server_base.cc:1034] running on GCE node
I20260927 22:42:42.633734 12927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:42:42.634649 12927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:42:42.635851 12927 hybrid_clock.cc:648] HybridClock initialized: now 1790548962635809 us; error 30 us; skew 500 ppm
I20260927 22:42:42.636145 12927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:42:42 dist-test-slave-pp80 krb5kdc[12089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:42:42.639350 12927 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:42:42.640897 12927 webserver.cc:460] Webserver started at http://127.1.53.67:33791/ using document root <none> and password file <none>
I20260927 22:42:42.641573 12927 fs_manager.cc:359] Metadata directory not provided
I20260927 22:42:42.641672 12927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:42:42.641932 12927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:42:42 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:42:42.656608 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "f5e5499509a34f4faad523e88df1010a"
format_stamp: "Formatted at 2026-09-27 22:42:42 on dist-test-slave-pp80"
server_key: "57113bc6f8883841c9e4e08f3912c147"
server_key_iv: "d8f4ad3a304c73044118d90e946774d6"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:42.657493 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "f5e5499509a34f4faad523e88df1010a"
format_stamp: "Formatted at 2026-09-27 22:42:42 on dist-test-slave-pp80"
server_key: "57113bc6f8883841c9e4e08f3912c147"
server_key_iv: "d8f4ad3a304c73044118d90e946774d6"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:42.663668 12927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:42:42.680547 12942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:42.682420 12927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:42:42.682623 12927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "f5e5499509a34f4faad523e88df1010a"
format_stamp: "Formatted at 2026-09-27 22:42:42 on dist-test-slave-pp80"
server_key: "57113bc6f8883841c9e4e08f3912c147"
server_key_iv: "d8f4ad3a304c73044118d90e946774d6"
server_key_version: "kuduclusterkey@0"
I20260927 22:42:42.682888 12927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:42:42.711027 12927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:42:42.716080 12927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:42:42.716434 12927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:42:42.717362 12927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:42:42.718684 12927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:42:42.719182 12927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:42.719280 12927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:42:42.719323 12927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:42:42.760812 12927 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:38029
I20260927 22:42:42.760848 13054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:38029 every 8 connection(s)
I20260927 22:42:42.762254 12927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:42:42.764289 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 12927
I20260927 22:42:42.764477 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:42:42.764999 1237 external_mini_cluster.cc:1428] Setting key 7d3b11ecd2a2126be3cecaa51338eb6d
Sep 27 22:42:42 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548962, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:42:42.784555 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.765663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:41665 (local address 127.1.53.126:37647)
0927 22:42:42.766491 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:42.766496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:42.767414 (+ 918us) server_negotiation.cc:407] Connection header received
0927 22:42:42.769856 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:42.769860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:42.769927 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:42.770064 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:42.771648 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.772498 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.774122 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.774530 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.774629 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.778158 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:42.778163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:42.778166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:42.778200 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:42.781631 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:42.782349 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:42.782353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:42.782356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:42.782492 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:42.782887 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:42.782890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:42.782892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:42.783167 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:42.783283 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:42.783955 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 22:42:42.784190 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":621,"threads_started":1}
I20260927 22:42:42.785565 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.766335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37647 (local address 127.1.53.67:41665)
0927 22:42:42.767246 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:42.767290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:42.769560 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:42.770284 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:42.770293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:42.770763 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:42:42.771422 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.771441 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.772645 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.772649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.773971 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.773989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.774743 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.774748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.774953 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.775699 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:42:42.775722 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:42:42.777916 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:42:42.781745 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:42:42.781753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:42:42.781767 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:42:42.782237 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:42:42.782589 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:42:42.782593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:42:42.782595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:42:42.782784 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:42:42.783283 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:42:42.783288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:42:42.783598 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 22:42:42.783826 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 22:42:42.784387 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":407,"threads_started":1}
I20260927 22:42:42.787066 13055 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37647
I20260927 22:42:42.787365 13055 heartbeater.cc:464] Registering TS with master...
I20260927 22:42:42.788045 13055 heartbeater.cc:511] Master 127.1.53.126:37647 requested a full tablet report, sending...
I20260927 22:42:42.789646 12565 ts_manager.cc:194] Registered new tserver with Master: f5e5499509a34f4faad523e88df1010a (127.1.53.67:38029)
I20260927 22:42:42.790414 12565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:41665
I20260927 22:42:42.793818 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:42:42.812465 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.799455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.1.53.126:37647)
0927 22:42:42.799636 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:42.799641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:42.800217 (+ 576us) server_negotiation.cc:407] Connection header received
0927 22:42:42.800455 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:42.800462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:42.800552 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:42.800661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:42.802162 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.802891 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.804300 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.804755 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.804898 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.807232 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:42.807240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:42.807250 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:42.807282 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:42.809948 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:42.810595 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:42.810601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:42.810605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:42.810773 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:42.811240 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:42.811243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:42.811245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:42.811509 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:42.811679 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:42.811991 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 22:42:42.812151 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:42:42.816610 12565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:42:42.824470 12565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:42:42.856005 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.838002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.1.53.65:32887)
0927 22:42:42.845092 (+ 7090us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:42.845105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:42.846672 (+ 1567us) server_negotiation.cc:407] Connection header received
0927 22:42:42.846780 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:42.846790 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:42.847085 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:42.847315 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:42:42.848789 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.850302 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.853239 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.854738 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.854874 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.855106 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:42.855488 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 22:42:42.855670 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6833,"thread_start_us":6577,"threads_started":1}
I20260927 22:42:42.856509 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.837719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32887 (local address 127.0.0.1:44382)
0927 22:42:42.846235 (+ 8516us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:42.846281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:42.846464 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:42.847645 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:42.847656 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:42.847696 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:42:42.848586 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.848602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.850444 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.850450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.853043 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.853058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.854980 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.854986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.855185 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.855243 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:42:42.855390 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:42:42.855580 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7507,"spinlock_wait_cycles":2816,"thread_start_us":7345,"threads_started":1}
I20260927 22:42:42.858053 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.837931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:37831 (local address 127.0.0.1:48422)
0927 22:42:42.846235 (+ 8304us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:42.846281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:42.846464 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:42.850577 (+ 4113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:42.850584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:42.850619 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:42:42.851054 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.851066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.852585 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.852591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.854859 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.854875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.857228 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.857238 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.857419 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.857457 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:42:42.857576 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:42:42.857732 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8127,"mutex_wait_us":100,"spinlock_wait_cycles":1792,"thread_start_us":7987,"threads_started":1}
I20260927 22:42:42.858251 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.838406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.1.53.66:37831)
0927 22:42:42.849978 (+ 11572us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:42.849999 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:42.850044 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:42:42.850140 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:42.850147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:42.850393 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:42.850579 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:42:42.851271 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.852392 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.855060 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.856950 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.857156 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.857385 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:42.857784 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 22:42:42.857955 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10247,"thread_start_us":10153,"threads_started":1}
I20260927 22:42:42.859695 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.846981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:38029 (local address 127.0.0.1:39580)
0927 22:42:42.849290 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:42.849321 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:42.849464 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:42.850843 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:42.850846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:42.850869 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:42:42.851310 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.851327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.853228 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.853232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.855310 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:42.855322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.858887 (+ 3565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.858895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:42.859063 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.859095 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:42:42.859200 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:42:42.859347 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2129,"thread_start_us":2957,"threads_started":1}
I20260927 22:42:42.859902 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:42.846730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.1.53.67:38029)
0927 22:42:42.850047 (+ 3317us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:42.850054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:42.850082 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:42:42.850155 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:42.850161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:42.850379 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:42.850581 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:42:42.851510 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.853070 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.855502 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:42.858661 (+ 3159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:42.858801 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:42.859032 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:42.859265 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:42:42.859508 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3164,"thread_start_us":3000,"threads_started":1}
I20260927 22:42:42.861502 12713 tablet_service.cc:1467] Processing CreateTablet for tablet 2980b411cebe42ceb77c980bccd7424d (DEFAULT_TABLE table=test-table [id=1efa82386ba94f1bbd16219fd5922041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:42:42.862095 12856 tablet_service.cc:1467] Processing CreateTablet for tablet 2980b411cebe42ceb77c980bccd7424d (DEFAULT_TABLE table=test-table [id=1efa82386ba94f1bbd16219fd5922041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:42:42.863211 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2980b411cebe42ceb77c980bccd7424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:42:42.863489 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2980b411cebe42ceb77c980bccd7424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:42:42.863366 12990 tablet_service.cc:1467] Processing CreateTablet for tablet 2980b411cebe42ceb77c980bccd7424d (DEFAULT_TABLE table=test-table [id=1efa82386ba94f1bbd16219fd5922041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:42:42.864717 12990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2980b411cebe42ceb77c980bccd7424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:42:42.874567 13074 tablet_bootstrap.cc:492] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Bootstrap starting.
I20260927 22:42:42.879331 13076 tablet_bootstrap.cc:492] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Bootstrap starting.
I20260927 22:42:42.880287 13074 tablet_bootstrap.cc:654] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Neither blocks nor log segments found. Creating new log.
I20260927 22:42:42.880942 13075 tablet_bootstrap.cc:492] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Bootstrap starting.
I20260927 22:42:42.881675 13074 log.cc:824] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Log is configured to *not* fsync() on all Append() calls
I20260927 22:42:42.883010 13076 tablet_bootstrap.cc:654] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Neither blocks nor log segments found. Creating new log.
I20260927 22:42:42.884899 13075 tablet_bootstrap.cc:654] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Neither blocks nor log segments found. Creating new log.
I20260927 22:42:42.885030 13076 log.cc:824] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:42:42.886210 13074 tablet_bootstrap.cc:492] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: No bootstrap required, opened a new log
I20260927 22:42:42.886206 13075 log.cc:824] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Log is configured to *not* fsync() on all Append() calls
I20260927 22:42:42.886531 13074 ts_tablet_manager.cc:1397] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260927 22:42:42.888155 13076 tablet_bootstrap.cc:492] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: No bootstrap required, opened a new log
I20260927 22:42:42.888432 13076 ts_tablet_manager.cc:1397] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 22:42:42.894464 13075 tablet_bootstrap.cc:492] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: No bootstrap required, opened a new log
I20260927 22:42:42.894767 13075 ts_tablet_manager.cc:1397] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 22:42:42.898089 13074 raft_consensus.cc:348] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:42.898597 13074 raft_consensus.cc:374] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:42:42.898742 13074 raft_consensus.cc:729] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e43bb1156d474da1a85ed77af08435, State: Initialized, Role: FOLLOWER
I20260927 22:42:42.899415 13074 consensus_queue.cc:260] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:42.899459 13076 raft_consensus.cc:348] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:42.899885 13076 raft_consensus.cc:374] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:42:42.900080 13076 raft_consensus.cc:729] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e5499509a34f4faad523e88df1010a, State: Initialized, Role: FOLLOWER
I20260927 22:42:42.900799 13076 consensus_queue.cc:260] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:42.901901 13055 heartbeater.cc:503] Master 127.1.53.126:37647 was elected leader, sending a full tablet report...
I20260927 22:42:42.902621 13076 ts_tablet_manager.cc:1428] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 22:42:42.902614 13074 ts_tablet_manager.cc:1428] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 22:42:42.905735 13075 raft_consensus.cc:348] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:42.906200 13075 raft_consensus.cc:374] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:42:42.906309 13075 raft_consensus.cc:729] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd01b29c2bc2482a87f3b0c154e86f82, State: Initialized, Role: FOLLOWER
I20260927 22:42:42.906920 13075 consensus_queue.cc:260] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:42.907935 12921 heartbeater.cc:503] Master 127.1.53.126:37647 was elected leader, sending a full tablet report...
I20260927 22:42:42.908564 13075 ts_tablet_manager.cc:1428] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260927 22:42:42.909570 12779 tablet.cc:2369] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:42:42.928118 12922 tablet.cc:2369] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:42:43.014366 13056 tablet.cc:2369] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:42:43.070910 13080 raft_consensus.cc:482] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:42:43.071264 13080 raft_consensus.cc:504] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:43.073079 13080 leader_election.cc:288] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd01b29c2bc2482a87f3b0c154e86f82 (127.1.53.66:37831), f5e5499509a34f4faad523e88df1010a (127.1.53.67:38029)
I20260927 22:42:43.080688 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.074432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:46693 (local address 127.1.53.67:38029)
0927 22:42:43.074671 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:43.074676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:43.074968 (+ 292us) server_negotiation.cc:407] Connection header received
0927 22:42:43.075202 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:43.075207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:43.075273 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:43.075372 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:42:43.076289 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.076975 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.078951 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.079736 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.079884 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.079987 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:43.080302 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:42:43.080423 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:42:43.080806 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.074260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:38029 (local address 127.1.53.65:46693)
0927 22:42:43.074855 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:43.074868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:43.075058 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:43.075392 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:43.075396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:43.075420 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:42:43.076116 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:43.076127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.077140 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.077144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:43.078795 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:43.078808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.079937 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.079943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:43.080124 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.080148 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:42:43.080292 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:42:43.080420 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":35,"thread_start_us":323,"threads_started":1}
I20260927 22:42:43.081415 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.073584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:59985 (local address 127.1.53.66:37831)
0927 22:42:43.073779 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:43.073784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:43.074273 (+ 489us) server_negotiation.cc:407] Connection header received
0927 22:42:43.074674 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:43.074680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:43.074787 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:43.074854 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:42:43.075563 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.076813 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.079016 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.079942 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.080109 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.080217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:43.080640 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:42:43.080772 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:42:43.081945 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.073406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:37831 (local address 127.1.53.65:59985)
0927 22:42:43.074159 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 22:42:43.074187 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:42:43.074433 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:42:43.075044 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:42:43.075047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:42:43.075070 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:42:43.075403 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:43.075411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.076954 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.076963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:43.078864 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:42:43.078877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.080182 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.080188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:42:43.080434 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.080471 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:42:43.081441 (+ 970us) client_negotiation.cc:240] Negotiation successful
0927 22:42:43.081574 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":51,"thread_start_us":385,"threads_started":1}
I20260927 22:42:43.082237 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2980b411cebe42ceb77c980bccd7424d" candidate_uuid: "19e43bb1156d474da1a85ed77af08435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e5499509a34f4faad523e88df1010a" is_pre_election: true
I20260927 22:42:43.082818 13010 raft_consensus.cc:2454] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e43bb1156d474da1a85ed77af08435 in term 0.
I20260927 22:42:43.083104 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2980b411cebe42ceb77c980bccd7424d" candidate_uuid: "19e43bb1156d474da1a85ed77af08435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" is_pre_election: true
I20260927 22:42:43.083540 12669 leader_election.cc:302] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e43bb1156d474da1a85ed77af08435, f5e5499509a34f4faad523e88df1010a; no voters:
I20260927 22:42:43.083688 12876 raft_consensus.cc:2454] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e43bb1156d474da1a85ed77af08435 in term 0.
I20260927 22:42:43.084061 13080 raft_consensus.cc:2781] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:42:43.084261 13080 raft_consensus.cc:482] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:42:43.084405 13080 raft_consensus.cc:3037] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:42:43.086701 13080 raft_consensus.cc:504] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:43.087253 13080 leader_election.cc:288] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [CANDIDATE]: Term 1 election: Requested vote from peers dd01b29c2bc2482a87f3b0c154e86f82 (127.1.53.66:37831), f5e5499509a34f4faad523e88df1010a (127.1.53.67:38029)
I20260927 22:42:43.087821 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2980b411cebe42ceb77c980bccd7424d" candidate_uuid: "19e43bb1156d474da1a85ed77af08435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd01b29c2bc2482a87f3b0c154e86f82"
I20260927 22:42:43.088440 12876 raft_consensus.cc:3037] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:42:43.088390 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2980b411cebe42ceb77c980bccd7424d" candidate_uuid: "19e43bb1156d474da1a85ed77af08435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e5499509a34f4faad523e88df1010a"
I20260927 22:42:43.088608 13010 raft_consensus.cc:3037] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:42:43.091611 13010 raft_consensus.cc:2454] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e43bb1156d474da1a85ed77af08435 in term 1.
I20260927 22:42:43.091609 12876 raft_consensus.cc:2454] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e43bb1156d474da1a85ed77af08435 in term 1.
I20260927 22:42:43.091990 12669 leader_election.cc:302] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e43bb1156d474da1a85ed77af08435, f5e5499509a34f4faad523e88df1010a; no voters:
I20260927 22:42:43.092348 13080 raft_consensus.cc:2781] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:42:43.092795 13080 raft_consensus.cc:686] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 1 LEADER]: Becoming Leader. State: Replica: 19e43bb1156d474da1a85ed77af08435, State: Running, Role: LEADER
I20260927 22:42:43.093552 13080 consensus_queue.cc:237] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } }
I20260927 22:42:43.098186 12565 catalog_manager.cc:5495] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e43bb1156d474da1a85ed77af08435 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "19e43bb1156d474da1a85ed77af08435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e43bb1156d474da1a85ed77af08435" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 } health_report { overall_health: UNKNOWN } } }
I20260927 22:42:43.153066 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.147183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.1.53.65:32887)
0927 22:42:43.147363 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:43.147368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:43.147384 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:42:43.147472 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:43.147476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:43.147557 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:43.147694 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:42:43.148109 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.148878 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.150338 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.151221 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.151328 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.151855 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:42:43.152519 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:42:43.152630 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:43.152751 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:42:43.152819 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:42:43.162040 13010 raft_consensus.cc:1261] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 1 FOLLOWER]: Refusing update from remote peer 19e43bb1156d474da1a85ed77af08435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:42:43.162123 12876 raft_consensus.cc:1261] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 1 FOLLOWER]: Refusing update from remote peer 19e43bb1156d474da1a85ed77af08435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:42:43.163116 13085 consensus_queue.cc:1035] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e5499509a34f4faad523e88df1010a" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:42:43.163484 13080 consensus_queue.cc:1035] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd01b29c2bc2482a87f3b0c154e86f82" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:42:43.197656 12564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:42:43.197996 12564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:42:43.201576 12564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 515d8ae583f14b2eac78dbb250130999: Sending DeleteTablet for 3 replicas of tablet 2980b411cebe42ceb77c980bccd7424d
I20260927 22:42:43.202728 12713 tablet_service.cc:1514] Processing DeleteTablet for tablet 2980b411cebe42ceb77c980bccd7424d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:42:43 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44382
I20260927 22:42:43.202935 12856 tablet_service.cc:1514] Processing DeleteTablet for tablet 2980b411cebe42ceb77c980bccd7424d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:42:43 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:48422
I20260927 22:42:43.203425 12990 tablet_service.cc:1514] Processing DeleteTablet for tablet 2980b411cebe42ceb77c980bccd7424d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:42:43 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:39580
I20260927 22:42:43.204840 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 12645
I20260927 22:42:43.211455 13097 tablet_replica.cc:330] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: stopping tablet replica
I20260927 22:42:43.216838 13097 raft_consensus.cc:2229] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:42:43.217779 13097 raft_consensus.cc:2258] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:42:43.218778 13099 tablet_replica.cc:330] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: stopping tablet replica
I20260927 22:42:43.219383 13099 raft_consensus.cc:2229] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:42:43.220077 13099 raft_consensus.cc:2258] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:42:43.221107 13097 ts_tablet_manager.cc:1905] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:42:43.222683 13099 ts_tablet_manager.cc:1905] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:42:43.232714 13099 ts_tablet_manager.cc:1918] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:42:43.232816 13098 tablet_replica.cc:330] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: stopping tablet replica
I20260927 22:42:43.232911 13099 log.cc:1196] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/2980b411cebe42ceb77c980bccd7424d
I20260927 22:42:43.233299 13098 raft_consensus.cc:2229] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:42:43.233428 13099 ts_tablet_manager.cc:1939] T 2980b411cebe42ceb77c980bccd7424d P f5e5499509a34f4faad523e88df1010a: Deleting consensus metadata
Sep 27 22:42:43 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:42:43.233844 13098 raft_consensus.cc:2258] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:42:43.235412 12552 catalog_manager.cc:4841] TS f5e5499509a34f4faad523e88df1010a (127.1.53.67:38029): tablet 2980b411cebe42ceb77c980bccd7424d (table test-table [id=1efa82386ba94f1bbd16219fd5922041]) successfully deleted
I20260927 22:42:43.235987 13098 ts_tablet_manager.cc:1905] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:42:43.237334 13097 ts_tablet_manager.cc:1918] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:42:43.237541 13097 log.cc:1196] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/2980b411cebe42ceb77c980bccd7424d
I20260927 22:42:43.238137 13097 ts_tablet_manager.cc:1939] T 2980b411cebe42ceb77c980bccd7424d P 19e43bb1156d474da1a85ed77af08435: Deleting consensus metadata
I20260927 22:42:43.240113 12550 catalog_manager.cc:4841] TS 19e43bb1156d474da1a85ed77af08435 (127.1.53.65:32887): tablet 2980b411cebe42ceb77c980bccd7424d (table test-table [id=1efa82386ba94f1bbd16219fd5922041]) successfully deleted
I20260927 22:42:43.243932 13098 ts_tablet_manager.cc:1918] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:42:43.244035 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.206318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.1.53.65:32887)
0927 22:42:43.206574 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:43.206583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:43.218773 (+ 12190us) server_negotiation.cc:407] Connection header received
0927 22:42:43.219077 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:43.219083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:43.219170 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:43.219271 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:43.221470 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.222628 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.228357 (+ 5729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.229071 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.229186 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.237008 (+ 7822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:43.237023 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:43.237041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:43.237085 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:43.241035 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:43.241870 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:43.241877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:43.241880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:43.242016 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:43.242416 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:43.242424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:43.242427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:43.242794 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:43.243102 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:43.243426 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:42:43.243655 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":94}
I20260927 22:42:43.244251 13098 log.cc:1196] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/2980b411cebe42ceb77c980bccd7424d
I20260927 22:42:43.244787 13098 ts_tablet_manager.cc:1939] T 2980b411cebe42ceb77c980bccd7424d P dd01b29c2bc2482a87f3b0c154e86f82: Deleting consensus metadata
I20260927 22:42:43.244853 12773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:42:43.246585 12552 catalog_manager.cc:4841] TS dd01b29c2bc2482a87f3b0c154e86f82 (127.1.53.66:37831): tablet 2980b411cebe42ceb77c980bccd7424d (table test-table [id=1efa82386ba94f1bbd16219fd5922041]) successfully deleted
I20260927 22:42:43.343056 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 12645
I20260927 22:42:43.361096 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 12784
Sep 27 22:42:43 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:42:43.376766 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.361973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.1.53.66:37831)
0927 22:42:43.362139 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:43.362143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:43.362161 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:42:43.362214 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:43.362217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:43.362290 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:43.362392 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:43.363628 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.364458 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.365726 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.366127 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.366262 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.370727 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:43.370742 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:43.370760 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:43.370805 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:43.373966 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:43.374599 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:43.374606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:43.374608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:43.374737 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:43.375150 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:43.375155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:43.375157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:43.375588 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:43.375844 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:43.376195 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:42:43.376459 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:42:43.377562 12916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:42:43.469094 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 12784
I20260927 22:42:43.485868 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 12927
Sep 27 22:42:43 dist-test-slave-pp80 krb5kdc[12089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:42:43.501017 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:42:43.486726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.1.53.67:38029)
0927 22:42:43.486894 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:42:43.486899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:42:43.486961 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:42:43.487123 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:42:43.487127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:42:43.487189 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:42:43.487299 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:42:43.488731 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.489537 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.490840 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:42:43.491200 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:42:43.491291 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:42:43.495585 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:42:43.495598 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:42:43.495615 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:42:43.495650 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:42:43.498372 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:43.499034 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:43.499041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:43.499043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:43.499160 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:42:43.499488 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:42:43.499491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:42:43.499493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:42:43.499838 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:42:43.500051 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:42:43.500518 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 22:42:43.500741 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:42:43.501665 13050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:42:43.592453 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 12927
I20260927 22:42:43.608788 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 12533
I20260927 22:42:43.610080 12595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:42:43.722522 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 12533
I20260927 22:42:43.746766 1237 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:42:44.082373 1237 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:42:44.082635 1237 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:42:44.428174 1237 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:42:44.428 UTC [12119] LOG: received smart shutdown request
2026-09-27 22:42:44.432 UTC [12119] LOG: background worker "logical replication launcher" (PID 12128) exited with exit code 1
2026-09-27 22:42:44.432 UTC [12123] LOG: shutting down
2026-09-27 22:42:44.626 UTC [12119] LOG: database system is shut down
2026-09-27T22:42:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50994 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:42:44 dist-test-slave-pp80 krb5kdc[13113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:42:44 dist-test-slave-pp80 krb5kdc[13113](info): set up 2 sockets
Sep 27 22:42:44 dist-test-slave-pp80 krb5kdc[13113](info): commencing operation
krb5kdc: starting...
W20260927 22:42:45.868644 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:42:46 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790548966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:42:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/rangeradmin_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/rangeradmin_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/rangerlookup_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/rangerlookup_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
I20260927 22:42:46.195657 1237 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkEfv30/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/postgres -l logfile start
2026-09-27 22:42:48.904 UTC [13143] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:42:48.904 UTC [13143] LOG: listening on IPv4 address "127.1.53.84", port 49261
2026-09-27 22:42:48.909 UTC [13143] LOG: listening on Unix socket "/tmp/.s.PGSQL.49261"
2026-09-27 22:42:48.931 UTC [13146] LOG: database system was shut down at 2026-09-27 22:42:47 UTC
2026-09-27 22:42:48.935 UTC [13143] LOG: database system is ready to accept connections
I20260927 22:42:49.908351 1237 mini_postgres.cc:96] Postgres bound to 49261
2026-09-27 22:42:49.925 UTC [13154] FATAL: database "slave" does not exist
127.1.53.84:49261 - accepting connections
I20260927 22:42:49.925788 1237 mini_ranger.cc:161] Starting Ranger...
I20260927 22:42:49.952024 1237 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:42:50.522306 1237 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:42:50.522421 1237 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-admin
2026-09-27 22:42:50,825 [I] DB FLAVOR :POSTGRES
2026-09-27 22:42:50,826 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:42:50,826 [I] Checking connection..
2026-09-27 22:42:50,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:42:51,059 [I] Checking connection passed.
2026-09-27 22:42:51,060 [I] --------- Verifying version history table ---------
2026-09-27 22:42:51,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:42:51,322 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:42:51,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:42:51,586 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:42:51,587 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:42:51,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:42:51.824 UTC [13218] WARNING: there is no transaction in progress
2026-09-27 22:42:51,831 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:42:51,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:42:52,078 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:42:52,078 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:42:52,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:42:52,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:42:52,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pp80.c.gcp-upstream.internal','N') ;"
2026-09-27 22:42:52,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:42:53,083 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:42:53,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:42:53,326 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:42:53,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:42:53,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:42:53,791 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:42:53,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:42:54.802 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.820 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.837 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.854 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.874 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.892 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.948 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.956 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.971 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.982 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:54.993 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:55.008 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:55.018 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:55.024 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:55.031 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:55.038 UTC [13326] WARNING: there is no transaction in progress
2026-09-27 22:42:55,759 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:42:55,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:42:56,005 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:42:56,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:42:56,273 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:42:56,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:42:56,538 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:42:56,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:42:56,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:42:57,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pp80.c.gcp-upstream.internal';"
2026-09-27 22:42:57,284 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:42:57,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:42:57,532 [I] DB_PATCHES have already been applied
I20260927 22:42:57.537971 1237 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkEfv30/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:42:57.538070 1237 mini_ranger.cc:193] Using host: 127.1.53.84
I20260927 22:42:57.552665 1237 mini_ranger.cc:227] Ranger admin URL: http://127.1.53.84:48075
Sep 27, 2026 10:42:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:42:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:42:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:42:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:42:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:42:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:42:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:42:58.710765 1237 mini_ranger.cc:160] Time spent starting Ranger: real 8.785s user 0.005s sys 0.033s
Sep 27, 2026 10:43:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:43:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:43:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:43:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:43:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:43:21.082046 1237 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/rangerkms_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/rangerkms_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.53.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.53.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.53.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/HTTP_127.1.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:43:21 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549001, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:43:21.267962 1237 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:43:21.288853 1237 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:43:22.302713 1237 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:43:22.302914 1237 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms
I20260927 22:43:22.303007 1237 mini_ranger_kms.cc:227] Using postgres at 127.1.53.84:49261
2026-09-27 22:43:22,672 [I] DB FLAVOR :POSTGRES
2026-09-27 22:43:22,672 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:43:22,673 [I] Checking connection
2026-09-27 22:43:22,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:43:22,919 [I] connection success
2026-09-27 22:43:22,919 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:43:22,919 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:43:22,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:43:23,163 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:43:23,163 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:43:23,163 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:43:23,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.53.84:49261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:43:23,471 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:43:23.988787 1237 mini_ranger_kms.cc:310] Created kms service
I20260927 22:43:24.176014 1237 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:43:24.220180 1237 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:43:24 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549001, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:24.665899 1237 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:43:24.666222 1237 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkEfv30/build/asan/bin/postgresql.jar:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkEfv30/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:43:24.666276 1237 mini_ranger_kms.cc:240] Using host: 127.1.53.84
I20260927 22:43:24.684764 1237 mini_ranger_kms.cc:276] Ranger KMS PID: 13507
I20260927 22:43:24.684897 1237 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.53.84:54107
Sep 27, 2026 10:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:43:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:43:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:43:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 22:43:26 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549006, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.53.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:26.987468 1237 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.719s user 0.079s sys 0.311s
I20260927 22:43:26.987737 1237 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:43:26.987946 1237 mini_ranger_kms.cc:401] 127.1.53.84:54107/kms/v1/keys
Sep 27 22:43:27 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549006, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.53.84@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
Sep 27 22:43:27 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549006, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.53.84@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
Sep 27 22:43:29 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:43:30.056954 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33919
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:33919
--ranger_config_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:30.398142 13557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:30.398468 13557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:30.398556 13557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:30.405819 13557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:30.405959 13557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:30.406011 13557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:30.406070 13557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:30.406116 13557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:30.416189 13557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.53.84:54107/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:33919
--ranger_config_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:30.417510 13557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:30.418788 13557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:30.431123 13562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:30.431027 13565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:30.431067 13563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:30.432216 13557 server_base.cc:1034] running on GCE node
I20260927 22:43:30.433027 13557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:30.434511 13557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:30.435743 13557 hybrid_clock.cc:648] HybridClock initialized: now 1790549010435688 us; error 53 us; skew 500 ppm
I20260927 22:43:30.436113 13557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:30 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:30.444656 13557 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:30.446518 13557 webserver.cc:460] Webserver started at http://127.1.53.126:33371/ using document root <none> and password file <none>
I20260927 22:43:30.447232 13557 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:30.447371 13557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:30.447728 13557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:43:30 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:30.499004 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "5a6e51a154f64dc8a2d63945040c3b94"
format_stamp: "Formatted at 2026-09-27 22:43:30 on dist-test-slave-pp80"
server_key: "a23abf6c153632b6cf6542465fef0485"
server_key_iv: "69f623940bd2237d9f823df59d3fb73d"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:30.499878 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "5a6e51a154f64dc8a2d63945040c3b94"
format_stamp: "Formatted at 2026-09-27 22:43:30 on dist-test-slave-pp80"
server_key: "a23abf6c153632b6cf6542465fef0485"
server_key_iv: "69f623940bd2237d9f823df59d3fb73d"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:30.506247 13557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 22:43:30.524149 13572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:30.525975 13557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:43:30.526194 13557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "5a6e51a154f64dc8a2d63945040c3b94"
format_stamp: "Formatted at 2026-09-27 22:43:30 on dist-test-slave-pp80"
server_key: "a23abf6c153632b6cf6542465fef0485"
server_key_iv: "69f623940bd2237d9f823df59d3fb73d"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:30.526423 13557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:30.573079 13557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:30.577831 13557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:30.578213 13557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:30.599939 13557 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:33919
I20260927 22:43:30.599967 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:33919 every 8 connection(s)
I20260927 22:43:30.601625 13557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:30.603276 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 13557
I20260927 22:43:30.603509 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:43:30.604182 1237 external_mini_cluster.cc:1428] Setting key 881095463f1c189ce54f686c75c52eaf
I20260927 22:43:30.609463 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:30.625507 13624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Bootstrap starting.
I20260927 22:43:30.629658 13624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:30.631096 13624 log.cc:824] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:43:30 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:30.634632 13624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: No bootstrap required, opened a new log
I20260927 22:43:30.642469 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:30.606354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.1.53.126:33919)
0927 22:43:30.617469 (+ 11115us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:30.617489 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:30.617936 (+ 447us) server_negotiation.cc:407] Connection header received
0927 22:43:30.620477 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:30.620508 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:30.621050 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:30.621557 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:30.623230 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:30.625324 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:30.627223 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:30.628308 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:30.628484 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:30.633476 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:30.633487 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:30.633507 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:30.633562 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:30.637631 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:30.638506 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:30.638514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:30.638523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:30.638740 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:30.639179 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:30.639184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:30.639187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:30.639922 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:30.640209 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:30.640713 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 22:43:30.641129 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10822,"thread_start_us":10563,"threads_started":1}
I20260927 22:43:30.643743 13624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:30.644266 13624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:30.644385 13624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6e51a154f64dc8a2d63945040c3b94, State: Initialized, Role: FOLLOWER
I20260927 22:43:30.645071 13624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:30.645345 13624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:43:30.645533 13624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:43:30.645753 13624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:30.647894 13624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:30.648514 13624 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6e51a154f64dc8a2d63945040c3b94; no voters:
I20260927 22:43:30.649205 13624 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:43:30.649324 13629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:30.650738 13629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 LEADER]: Becoming Leader. State: Replica: 5a6e51a154f64dc8a2d63945040c3b94, State: Running, Role: LEADER
I20260927 22:43:30.651366 13629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:30.651538 13624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:30.654213 13631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6e51a154f64dc8a2d63945040c3b94. Latest consensus state: current_term: 1 leader_uuid: "5a6e51a154f64dc8a2d63945040c3b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } } }
I20260927 22:43:30.654593 13631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:30.655109 13630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a6e51a154f64dc8a2d63945040c3b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } } }
I20260927 22:43:30.655392 13630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:30.660725 13635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:43:30.668665 13635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:43:30.674768 13624 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:43:30.692340 13635 catalog_manager.cc:1347] Generated new cluster ID: 5ca29a9c36814a42b74a8a21d8fc3668
I20260927 22:43:30.692663 13635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:43:30.714057 13635 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:43:30.715986 13635 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:43:30.727948 13635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Generated new TSK 0
I20260927 22:43:30.728989 13635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:43:31.504464 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33919
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:43:31 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:43:31.906450 13669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:31.906785 13669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:31.906947 13669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:31.914389 13669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:31.914559 13669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:31.914731 13669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:43:31.927789 13669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.53.84:54107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:31.929502 13669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:31.931272 13669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:31.951761 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:31.952544 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:31.955967 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:31.956430 13669 server_base.cc:1034] running on GCE node
I20260927 22:43:31.957520 13669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:31.960542 13669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:31.961838 13669 hybrid_clock.cc:648] HybridClock initialized: now 1790549011961783 us; error 57 us; skew 500 ppm
I20260927 22:43:31.962270 13669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:31 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:31.967382 13669 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:43:31.971870 13669 webserver.cc:460] Webserver started at http://127.1.53.65:37853/ using document root <none> and password file <none>
I20260927 22:43:31.972894 13669 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:31.973054 13669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:31.973402 13669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:43:31 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:31.991966 13669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "6305f305ab0843e1a0de6af543c6cbce"
format_stamp: "Formatted at 2026-09-27 22:43:31 on dist-test-slave-pp80"
server_key: "690fc0a94455ba1f8df05a62fe197189"
server_key_iv: "c7eee2db6949f72ab008b588bfc1f3f1"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:31.993209 13669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "6305f305ab0843e1a0de6af543c6cbce"
format_stamp: "Formatted at 2026-09-27 22:43:31 on dist-test-slave-pp80"
server_key: "690fc0a94455ba1f8df05a62fe197189"
server_key_iv: "c7eee2db6949f72ab008b588bfc1f3f1"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:32.014297 13669 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.015s sys 0.000s
I20260927 22:43:32.036351 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:32.038419 13669 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:43:32.038708 13669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "6305f305ab0843e1a0de6af543c6cbce"
format_stamp: "Formatted at 2026-09-27 22:43:31 on dist-test-slave-pp80"
server_key: "690fc0a94455ba1f8df05a62fe197189"
server_key_iv: "c7eee2db6949f72ab008b588bfc1f3f1"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:32.039100 13669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:32.074438 13669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:32.080096 13669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:32.080453 13669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:32.081662 13669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:32.088479 13669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:32.088649 13669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:32.088836 13669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:32.088949 13669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:43:32 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:32.467092 13669 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:32827
I20260927 22:43:32.467175 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:32827 every 8 connection(s)
I20260927 22:43:32.468816 13669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:43:32.472287 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 13669
I20260927 22:43:32.472503 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:43:32.473222 1237 external_mini_cluster.cc:1428] Setting key 4325ea836e7f9035a7da7048d4335ba3
Sep 27 22:43:32 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:32.516285 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:32.473421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:46029 (local address 127.1.53.126:33919)
0927 22:43:32.484449 (+ 11028us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:32.484455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:32.484474 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:43:32.484535 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:32.484541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:32.484624 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:32.484725 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:32.486855 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:32.487996 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:32.490735 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:32.491415 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:32.491538 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:32.508010 (+ 16472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:32.508020 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:32.508026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:32.508090 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:32.512136 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:32.513292 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:32.513299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:32.513303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:32.513463 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:32.514052 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:32.514057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:32.514060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:32.514407 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:32.514540 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:32.515594 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0927 22:43:32.515824 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10899,"thread_start_us":10727,"threads_started":1}
I20260927 22:43:32.518923 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:32.472749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33919 (local address 127.1.53.65:46029)
0927 22:43:32.480907 (+ 8158us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:32.480985 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:32.483693 (+ 2708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:32.485173 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:32.485185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:32.485794 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:32.486644 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:32.486671 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:32.488213 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:32.488218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:32.490539 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:32.490565 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:32.491917 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:32.491924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:32.492271 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:32.493150 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:32.493180 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:32.507584 (+ 14404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:32.512429 (+ 4845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:32.512440 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:32.512462 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:32.513128 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:32.513627 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:32.513632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:32.513636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:32.513905 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:32.514634 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:32.514642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:32.515127 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 22:43:32.516788 (+ 1661us) client_negotiation.cc:240] Negotiation successful
0927 22:43:32.517410 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7792,"thread_start_us":7542,"threads_started":1}
I20260927 22:43:32.521332 13802 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33919
I20260927 22:43:32.521660 13802 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:32.522567 13802 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:32.525342 13589 ts_manager.cc:194] Registered new tserver with Master: 6305f305ab0843e1a0de6af543c6cbce (127.1.53.65:32827)
I20260927 22:43:32.527482 13589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:46029
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:43:32.576488 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33919
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:43:32 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
W20260927 22:43:33.062377 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:33.062713 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:33.062888 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:33.070385 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:33.070546 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:33.070739 13808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:43:33.080900 13808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.53.84:54107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:33.082271 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:33.083622 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:33.101194 13822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:33.105029 13823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:33.108132 13825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:33.109257 13808 server_base.cc:1034] running on GCE node
I20260927 22:43:33.109956 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:33.111088 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:33.112345 13808 hybrid_clock.cc:648] HybridClock initialized: now 1790549013112275 us; error 61 us; skew 500 ppm
I20260927 22:43:33.112699 13808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549013, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:33.115936 13808 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:43:33.117717 13808 webserver.cc:460] Webserver started at http://127.1.53.66:42347/ using document root <none> and password file <none>
I20260927 22:43:33.118449 13808 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:33.118594 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:33.118908 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549013, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:33.136655 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "851b3026588841b68de9ec3b9168150b"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "5a4b2c724eea628835da52692a8934f2"
server_key_iv: "233ad7761581363d3079327bc5e0ed6b"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:33.137722 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "851b3026588841b68de9ec3b9168150b"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "5a4b2c724eea628835da52692a8934f2"
server_key_iv: "233ad7761581363d3079327bc5e0ed6b"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:33.144956 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 22:43:33.164196 13832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:33.166216 13808 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:43:33.166410 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "851b3026588841b68de9ec3b9168150b"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "5a4b2c724eea628835da52692a8934f2"
server_key_iv: "233ad7761581363d3079327bc5e0ed6b"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:33.166671 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:33.197922 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:33.203262 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:33.203785 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:33.205098 13808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:33.206844 13808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:33.206918 13808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:33.207063 13808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:33.207123 13808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:33.244263 13808 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:46467
I20260927 22:43:33.244297 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:46467 every 8 connection(s)
I20260927 22:43:33.245715 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:43:33.250504 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 13808
I20260927 22:43:33.250697 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:43:33.251242 1237 external_mini_cluster.cc:1428] Setting key 7061065864c048a21ff0784300a31ed8
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549013, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:33.269999 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:33.249279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:57473 (local address 127.1.53.126:33919)
0927 22:43:33.249772 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:33.249779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:33.251097 (+ 1318us) server_negotiation.cc:407] Connection header received
0927 22:43:33.253726 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:33.253732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:33.253816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:33.253941 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:33.255807 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.257070 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.259378 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.260195 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.260386 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:33.263947 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:33.263954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:33.263959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:33.264003 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:33.266767 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.267554 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.267558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.267560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.267676 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.268139 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.268145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.268148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.268489 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:33.268636 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:33.269344 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 22:43:33.269587 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260927 22:43:33.271183 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:33.249978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33919 (local address 127.1.53.66:57473)
0927 22:43:33.250928 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:33.250975 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:33.253400 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:33.254218 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:33.254230 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:33.254809 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:33.255602 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:33.255626 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.257236 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.257241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:33.259218 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:33.259241 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.260653 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.260658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:33.260879 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:33.261639 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:33.261668 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:33.263713 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:33.266916 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:33.266926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:33.266945 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:33.267373 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:33.267798 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:33.267802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:33.267806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:33.268022 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:33.268624 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:33.268633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:33.268975 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 22:43:33.269262 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 22:43:33.269799 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":420,"threads_started":1}
I20260927 22:43:33.272818 13945 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33919
I20260927 22:43:33.273162 13945 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:33.273896 13945 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:33.275393 13589 ts_manager.cc:194] Registered new tserver with Master: 851b3026588841b68de9ec3b9168150b (127.1.53.66:46467)
I20260927 22:43:33.276425 13589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:57473
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:43:33.313433 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33919
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:33.531737 13802 heartbeater.cc:503] Master 127.1.53.126:33919 was elected leader, sending a full tablet report...
W20260927 22:43:33.655037 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:33.655332 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:33.655432 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:33.663522 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:33.663636 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:33.663759 13951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:43:33.674659 13951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.53.84:54107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:33919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:33.675961 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:33.677423 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:33.691267 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:33.691285 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:33.691267 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:33.691918 13951 server_base.cc:1034] running on GCE node
I20260927 22:43:33.692728 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:33.693786 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:33.695075 13951 hybrid_clock.cc:648] HybridClock initialized: now 1790549013695034 us; error 90 us; skew 500 ppm
I20260927 22:43:33.695458 13951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549013, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:33.698913 13951 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:43:33.700495 13951 webserver.cc:460] Webserver started at http://127.1.53.67:38265/ using document root <none> and password file <none>
I20260927 22:43:33.701188 13951 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:33.701294 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:33.701576 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549013, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:33.720121 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "e62c96ba453644ea85ceccfe90db36a9"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "74fe8622e3915d2056fa407505991cf2"
server_key_iv: "424249dfd67bb9b346feccd016ca278c"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:33.721238 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "e62c96ba453644ea85ceccfe90db36a9"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "74fe8622e3915d2056fa407505991cf2"
server_key_iv: "424249dfd67bb9b346feccd016ca278c"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:33.728458 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:43:33.744601 13966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:33.746484 13951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:43:33.746692 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "e62c96ba453644ea85ceccfe90db36a9"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "74fe8622e3915d2056fa407505991cf2"
server_key_iv: "424249dfd67bb9b346feccd016ca278c"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:33.746954 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:33.761581 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:33.766141 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:33.766481 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:33.767500 13951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:33.769014 13951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:33.769095 13951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:33.769261 13951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:33.769312 13951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:33.804136 13951 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:37887
I20260927 22:43:33.804174 14078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:37887 every 8 connection(s)
I20260927 22:43:33.805542 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:43:33.810002 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 13951
I20260927 22:43:33.810210 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:43:33.810694 1237 external_mini_cluster.cc:1428] Setting key 5ed4ac08c9bb770a7cd06a5f2fb336d8
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549013, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:33.827903 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:33.808295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:53567 (local address 127.1.53.126:33919)
0927 22:43:33.808862 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:33.808867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:33.809933 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 22:43:33.812140 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:33.812149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:33.812306 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:33.812441 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:33.814357 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.815215 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.816955 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.817435 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.817554 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:33.820940 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:33.820945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:33.820949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:33.820982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:33.824111 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.825015 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.825022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.825025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.825199 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.825851 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.825908 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.825912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.826320 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:33.826448 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:33.827375 (+ 927us) server_negotiation.cc:299] Negotiation successful
0927 22:43:33.827590 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":251,"threads_started":1}
I20260927 22:43:33.828779 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:33.808925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33919 (local address 127.1.53.67:53567)
0927 22:43:33.809736 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:33.809781 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:33.811727 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:33.812652 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:33.812661 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:33.813311 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:33.814180 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:33.814201 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.815402 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.815406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:33.816761 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:33.816773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.817656 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.817661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:33.817860 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:33.818545 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:33.818568 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:33.820702 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:33.824313 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:33.824319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:33.824333 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:33.824851 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:33.825350 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:33.825356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:33.825360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:33.825679 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:33.826472 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:33.826479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:33.826944 (+ 465us) client_negotiation.cc:769] Sending connection context
0927 22:43:33.827224 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 22:43:33.827601 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260927 22:43:33.830420 14079 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33919
I20260927 22:43:33.830745 14079 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:33.831514 14079 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:33.833079 13589 ts_manager.cc:194] Registered new tserver with Master: e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887)
I20260927 22:43:33.833981 13589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:53567
I20260927 22:43:33.839380 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:43:33.841261 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 13669
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:43:33.859826 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:33.842248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.1.53.65:32827)
0927 22:43:33.842993 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:33.843024 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:33.843051 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:43:33.843114 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:33.843121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:33.843328 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:33.843469 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:33.845219 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.846601 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.848235 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.848931 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.849063 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:33.853875 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:33.853886 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:33.853900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:33.853934 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:33.856949 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.857638 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.857645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.857647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.857786 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.858135 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.858138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.858140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.858473 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:33.858696 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:33.859181 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 22:43:33.859482 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":398,"threads_started":1}
I20260927 22:43:33.861256 13797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:33.945708 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 13669
I20260927 22:43:33.961975 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 13808
Sep 27 22:43:33 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:43:33.979012 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:33.962928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.1.53.66:46467)
0927 22:43:33.963448 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:33.963455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:33.963479 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:43:33.963547 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:33.963553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:33.963771 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:33.964050 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:33.965351 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.966326 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.967664 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:33.968219 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:33.968420 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:33.972932 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:33.972944 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:33.972959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:33.972993 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:33.975738 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.976578 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.976592 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.976594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.976728 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:33.977165 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:33.977175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:33.977178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:33.977654 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:33.977998 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:33.978330 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:43:33.978638 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":275,"threads_started":1}
I20260927 22:43:33.980338 13940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:34.064306 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 13808
I20260927 22:43:34.083971 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 13951
Sep 27 22:43:34 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:43:34.100755 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:34.085104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.1.53.67:37887)
0927 22:43:34.085755 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:34.085762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:34.085789 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:43:34.085861 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:34.085867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:34.086080 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:34.086265 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:34.087607 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:34.088658 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:34.089976 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:34.090525 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:34.090673 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:34.094940 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:34.094950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:34.094965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:34.095000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:34.097916 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:34.098648 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:34.098656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:34.098658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:34.098784 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:34.099147 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:34.099152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:34.099156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:34.099485 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:34.099812 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:34.100109 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:43:34.100393 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":346,"threads_started":1}
I20260927 22:43:34.102056 14074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:34.185225 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 13951
I20260927 22:43:34.200718 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 13557
I20260927 22:43:34.201992 13619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:34.314007 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 13557
I20260927 22:43:34.338361 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33919
--webserver_interface=127.1.53.126
--webserver_port=33371
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:33919
--ranger_config_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:34.659006 14091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:34.659360 14091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:34.659521 14091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:34.666728 14091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:34.666833 14091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:34.666879 14091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:34.666962 14091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:34.667037 14091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:34.676911 14091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:33919
--ranger_config_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=33371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:34.678215 14091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:34.679809 14091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:34.691787 14096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:34.691742 14097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:34.692106 14099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:34.692657 14091 server_base.cc:1034] running on GCE node
I20260927 22:43:34.693424 14091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:34.694784 14091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:34.696013 14091 hybrid_clock.cc:648] HybridClock initialized: now 1790549014695864 us; error 162 us; skew 500 ppm
I20260927 22:43:34.696432 14091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:34 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:34.699684 14091 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:34.701299 14091 webserver.cc:460] Webserver started at http://127.1.53.126:33371/ using document root <none> and password file <none>
I20260927 22:43:34.701970 14091 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:34.702124 14091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:43:34 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:34.762027 14091 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:43:34.765298 14107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:34.766891 14091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:43:34.767153 14091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "5a6e51a154f64dc8a2d63945040c3b94"
format_stamp: "Formatted at 2026-09-27 22:43:30 on dist-test-slave-pp80"
server_key: "a23abf6c153632b6cf6542465fef0485"
server_key_iv: "69f623940bd2237d9f823df59d3fb73d"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:34.768160 14091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:34.791780 14091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:34.795714 14091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:34.796020 14091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:34.815162 14091 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:33919
I20260927 22:43:34.815186 14158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:33919 every 8 connection(s)
I20260927 22:43:34.816860 14091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:34.820780 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14091
I20260927 22:43:34.821607 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:32827
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:33919
--webserver_port=37853
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:34.824222 14159 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:43:34.828187 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Bootstrap starting.
I20260927 22:43:34.847146 14159 log.cc:824] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:34.856113 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:43:34.856709 14159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Bootstrap complete.
I20260927 22:43:34.865629 14159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:34.866498 14159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6e51a154f64dc8a2d63945040c3b94, State: Initialized, Role: FOLLOWER
I20260927 22:43:34.867228 14159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:34.867462 14159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:43:34.867586 14159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:43:34.867780 14159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:43:34.871630 14159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:34.872148 14159 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6e51a154f64dc8a2d63945040c3b94; no voters:
I20260927 22:43:34.872900 14159 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:43:34.873172 14163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:43:34.874193 14163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [term 2 LEADER]: Becoming Leader. State: Replica: 5a6e51a154f64dc8a2d63945040c3b94, State: Running, Role: LEADER
I20260927 22:43:34.874987 14159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:34.875025 14163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } }
I20260927 22:43:34.884626 14159 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:43:34.884524 14164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5a6e51a154f64dc8a2d63945040c3b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } } }
I20260927 22:43:34.885167 14164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:34.884853 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6e51a154f64dc8a2d63945040c3b94. Latest consensus state: current_term: 2 leader_uuid: "5a6e51a154f64dc8a2d63945040c3b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6e51a154f64dc8a2d63945040c3b94" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 33919 } } }
I20260927 22:43:34.885839 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94 [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:34.888092 14178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:43:34.900796 14178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:43:34.902560 14178 catalog_manager.cc:1259] Loaded cluster ID: 5ca29a9c36814a42b74a8a21d8fc3668
I20260927 22:43:34.902740 14178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:43:34.907225 14178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:43:34.909075 14178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Loaded TSK: 0
I20260927 22:43:34.910755 14178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:43:35.304351 14161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:35.304868 14161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:35.305086 14161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:35.317042 14161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:35.317206 14161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:35.317461 14161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:43:35.334162 14161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:32827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=37853
--tserver_master_addrs=127.1.53.126:33919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:35.336407 14161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:35.338367 14161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:35.358310 14196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:35.358235 14197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:35.359134 14199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:35.359997 14161 server_base.cc:1034] running on GCE node
I20260927 22:43:35.360769 14161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:35.363863 14161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:35.365130 14161 hybrid_clock.cc:648] HybridClock initialized: now 1790549015365055 us; error 77 us; skew 500 ppm
I20260927 22:43:35.365466 14161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:35 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:35.369902 14161 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:43:35.372004 14161 webserver.cc:460] Webserver started at http://127.1.53.65:37853/ using document root <none> and password file <none>
I20260927 22:43:35.373023 14161 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:35.373191 14161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:43:35 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:35.430644 14161 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.010s sys 0.007s
I20260927 22:43:35.436995 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:35.439069 14161 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 22:43:35.439299 14161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "6305f305ab0843e1a0de6af543c6cbce"
format_stamp: "Formatted at 2026-09-27 22:43:31 on dist-test-slave-pp80"
server_key: "690fc0a94455ba1f8df05a62fe197189"
server_key_iv: "c7eee2db6949f72ab008b588bfc1f3f1"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:35.440853 14161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:35.472059 14161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:35.477800 14161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:35.478209 14161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:35.479538 14161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:35.481302 14161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:35.481438 14161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:35.481549 14161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:35.481645 14161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:35.530774 14161 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:32827
I20260927 22:43:35.530824 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:32827 every 8 connection(s)
I20260927 22:43:35.532568 14161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:43:35.542098 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14161
I20260927 22:43:35.542817 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:46467
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:33919
--webserver_port=42347
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:43:35 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549015, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:35.582157 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:35.535749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33919 (local address 127.1.53.65:60465)
0927 22:43:35.551629 (+ 15880us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:35.551692 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:35.554289 (+ 2597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:35.556759 (+ 2470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:35.556773 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:35.557475 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:35.558329 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:35.558360 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:35.560960 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:35.560969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:35.563290 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:35.563311 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:35.565337 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:35.565348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:35.565656 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:35.566544 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:35.566583 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:35.569650 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:35.576415 (+ 6765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:35.576426 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:35.576444 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:35.577110 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:35.577704 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:35.577710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:35.577713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:35.578010 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:35.579216 (+ 1206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:35.579225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:35.579655 (+ 430us) client_negotiation.cc:769] Sending connection context
0927 22:43:35.579994 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 22:43:35.580628 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15574,"thread_start_us":15394,"threads_started":1}
I20260927 22:43:35.584750 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:35.535633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:60465 (local address 127.1.53.126:33919)
0927 22:43:35.550924 (+ 15291us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:35.550954 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:35.551886 (+ 932us) server_negotiation.cc:407] Connection header received
0927 22:43:35.555023 (+ 3137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:35.555046 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:35.555564 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:35.555997 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:35.558630 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:35.560735 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:35.563501 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:35.564695 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:35.564898 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:35.570061 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:35.570076 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:35.570104 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:35.570165 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:35.576128 (+ 5963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:35.577307 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:35.577318 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:35.577327 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:35.577541 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:35.578180 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:35.578186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:35.578192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:35.578924 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:35.579126 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:35.582887 (+ 3761us) server_negotiation.cc:299] Negotiation successful
0927 22:43:35.583373 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14860,"thread_start_us":16373,"threads_started":1}
I20260927 22:43:35.589097 14321 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33919
I20260927 22:43:35.589440 14321 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:35.590306 14321 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
W20260927 22:43:35.593559 14321 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:33919 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:43:35.593786 14321 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:35.594056 14321 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:35.595458 14321 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:35.595840 14321 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
W20260927 22:43:35.597066 14321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:43:36.006320 14325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:36.006662 14325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:36.006843 14325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:36.014850 14325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:36.015005 14325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:36.015179 14325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:43:36.028674 14325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:46467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=42347
--tserver_master_addrs=127.1.53.126:33919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:36.030068 14325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:36.031389 14325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:36.049839 14340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:36.049916 14339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:36.056212 14342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:36.057114 14325 server_base.cc:1034] running on GCE node
I20260927 22:43:36.058007 14325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:36.059402 14325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:36.060739 14325 hybrid_clock.cc:648] HybridClock initialized: now 1790549016060662 us; error 57 us; skew 500 ppm
I20260927 22:43:36.061158 14325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:36.065639 14325 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:43:36.067704 14325 webserver.cc:460] Webserver started at http://127.1.53.66:42347/ using document root <none> and password file <none>
I20260927 22:43:36.068670 14325 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:36.068830 14325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:36.090955 14325 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:43:36.108899 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:36.110952 14325 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.012s
I20260927 22:43:36.111218 14325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "851b3026588841b68de9ec3b9168150b"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "5a4b2c724eea628835da52692a8934f2"
server_key_iv: "233ad7761581363d3079327bc5e0ed6b"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:36.112792 14325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:36.149694 14325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:36.155336 14325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:36.155714 14325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:36.160984 14325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:36.162623 14325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:36.162746 14325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:36.162891 14325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:36.162990 14325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:36.209566 14325 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:46467
I20260927 22:43:36.209611 14464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:46467 every 8 connection(s)
I20260927 22:43:36.211330 14325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:43:36.220908 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14325
I20260927 22:43:36.221721 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.53.84:54107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:37887
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:33919
--webserver_port=38265
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:37677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:36.256505 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.214655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:39763 (local address 127.1.53.126:33919)
0927 22:43:36.221321 (+ 6666us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:36.221326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:36.232039 (+ 10713us) server_negotiation.cc:407] Connection header received
0927 22:43:36.237245 (+ 5206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:36.237251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:36.237359 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:36.237493 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:36.239873 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.241050 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.243558 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.244212 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.244348 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:36.248865 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:36.248873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:36.248878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:36.248920 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:36.252532 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.253527 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.253535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.253539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.253688 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.254287 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.254292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.254296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.254667 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:36.254797 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:36.255831 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0927 22:43:36.256098 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6542,"thread_start_us":6415,"threads_started":1}
I20260927 22:43:36.259078 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.215306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33919 (local address 127.1.53.66:39763)
0927 22:43:36.231764 (+ 16458us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:36.231855 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:36.236800 (+ 4945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:36.237973 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:36.237987 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:36.238679 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:36.239614 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:36.239662 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.241284 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.241289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:36.243360 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:36.243375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.244722 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.244729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:36.245036 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:36.245873 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:36.245909 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:36.248585 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:36.252712 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:36.252722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:36.252741 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:36.253362 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:36.253854 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:36.253858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:36.253861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:36.254143 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:36.254917 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:36.254925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:36.255341 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 22:43:36.257054 (+ 1713us) client_negotiation.cc:240] Negotiation successful
0927 22:43:36.257647 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15961,"thread_start_us":16843,"threads_started":1}
I20260927 22:43:36.261243 14465 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33919
I20260927 22:43:36.261569 14465 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:36.262420 14465 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:36.264619 14124 ts_manager.cc:194] Registered new tserver with Master: 851b3026588841b68de9ec3b9168150b (127.1.53.66:46467)
I20260927 22:43:36.266865 14124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:39763
I20260927 22:43:36.597244 14321 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:36.597692 14321 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:36.599181 14124 ts_manager.cc:194] Registered new tserver with Master: 6305f305ab0843e1a0de6af543c6cbce (127.1.53.65:32827)
I20260927 22:43:36.600607 14124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:60465
W20260927 22:43:36.638935 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:36.639278 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:36.639458 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:36.649627 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:36.649791 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:36.649982 14469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:43:36.663393 14469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.53.84:54107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:37887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=38265
--tserver_master_addrs=127.1.53.126:33919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:36.664930 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:36.666594 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:36.685791 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:36.687449 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:36.694691 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:36.696399 14469 server_base.cc:1034] running on GCE node
I20260927 22:43:36.697285 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:36.699970 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:36.701272 14469 hybrid_clock.cc:648] HybridClock initialized: now 1790549016701197 us; error 73 us; skew 500 ppm
I20260927 22:43:36.701632 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:36.705255 14469 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:43:36.711956 14469 webserver.cc:460] Webserver started at http://127.1.53.67:38265/ using document root <none> and password file <none>
I20260927 22:43:36.712896 14469 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:36.713078 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:36.746644 14469 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.007s
I20260927 22:43:36.763260 14486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:36.765811 14469 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.001s
I20260927 22:43:36.766134 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "e62c96ba453644ea85ceccfe90db36a9"
format_stamp: "Formatted at 2026-09-27 22:43:33 on dist-test-slave-pp80"
server_key: "74fe8622e3915d2056fa407505991cf2"
server_key_iv: "424249dfd67bb9b346feccd016ca278c"
server_key_version: "kuduclusterkey@0"
I20260927 22:43:36.767756 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:36.827648 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:36.831856 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:36.832149 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:36.833280 14469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:36.834764 14469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:36.834882 14469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:36.835023 14469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:36.835114 14469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:36.885811 14469 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:37887
I20260927 22:43:36.885905 14598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:37887 every 8 connection(s)
I20260927 22:43:36.887692 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:43:36.888309 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14469
Sep 27 22:43:36 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:36.921074 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.893498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:60083 (local address 127.1.53.126:33919)
0927 22:43:36.900741 (+ 7243us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:36.900751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:36.900776 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:43:36.900866 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:36.900871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:36.900966 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:36.901082 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:36.903169 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.904331 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.906701 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.907393 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.907496 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:36.911941 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:36.911948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:36.911953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:36.912044 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:36.915938 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.916894 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.916900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.916904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.917039 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.918320 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.918325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.918329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.918669 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:36.918799 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:36.920476 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0927 22:43:36.920697 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7064,"thread_start_us":6879,"threads_started":1}
I20260927 22:43:36.922338 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.892938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:33919 (local address 127.1.53.67:60083)
0927 22:43:36.895327 (+ 2389us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:36.895375 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:36.897698 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:36.901526 (+ 3828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:36.901538 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:36.902164 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:36.902966 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:36.902991 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.904523 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.904528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:36.906537 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:36.906553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.907882 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.907889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:36.908168 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:36.908968 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:36.908996 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:36.911638 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:36.916110 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:36.916120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:36.916139 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:36.916740 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:36.917831 (+ 1091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:36.917836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:36.917840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:36.918186 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:36.919295 (+ 1109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:36.919304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:36.919750 (+ 446us) client_negotiation.cc:769] Sending connection context
0927 22:43:36.920067 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 22:43:36.920660 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"thread_start_us":1885,"threads_started":1}
I20260927 22:43:36.922427 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.889423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.1.53.126:33919)
0927 22:43:36.889933 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:36.889940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:36.901199 (+ 11259us) server_negotiation.cc:407] Connection header received
0927 22:43:36.901400 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:36.901408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:36.901477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:36.901596 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:36.903707 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.904869 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.907032 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.907702 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.907805 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:36.911688 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:36.911696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:36.911701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:36.911739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:36.916205 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.917804 (+ 1599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.917809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.917813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.917984 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.919234 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.919240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.919243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.919640 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:36.919899 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:36.921838 (+ 1939us) server_negotiation.cc:299] Negotiation successful
0927 22:43:36.922056 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":268,"threads_started":1}
I20260927 22:43:36.924639 14599 heartbeater.cc:347] Connected to a master server at 127.1.53.126:33919
I20260927 22:43:36.924985 14599 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:36.925814 14599 heartbeater.cc:511] Master 127.1.53.126:33919 requested a full tablet report, sending...
I20260927 22:43:36.928282 14123 ts_manager.cc:194] Registered new tserver with Master: e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887)
I20260927 22:43:36.929523 14123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:60083
I20260927 22:43:36.940966 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:43:36.965276 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.944569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.1.53.126:33919)
0927 22:43:36.944746 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:36.944752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:36.948841 (+ 4089us) server_negotiation.cc:407] Connection header received
0927 22:43:36.949051 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:36.949060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:36.949168 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:36.949293 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:36.951407 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.952440 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.954380 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:36.955006 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:36.955126 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:36.958474 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:36.958482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:36.958487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:36.958537 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:36.962232 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.963141 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.963147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.963150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.963313 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:36.963734 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:36.963741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:36.963744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:36.964089 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:36.964391 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:36.964698 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:43:36.964939 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:43:36.971781 14123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:43:36.982750 14123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:43:37.027962 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.000280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:56954)
0927 22:43:37.012893 (+ 12613us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.012913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.013027 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.018699 (+ 5672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.018711 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.018759 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.019385 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.019402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.022411 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.022420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.025087 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.025102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.027011 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.027018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.027215 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.027271 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.027385 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.027587 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12438,"thread_start_us":11449,"threads_started":1}
I20260927 22:43:37.030177 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.000400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.1.53.65:32827)
0927 22:43:37.013619 (+ 13219us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.013633 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.013671 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:43:37.013797 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.013810 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.018369 (+ 4559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.019704 (+ 1335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.020062 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.022196 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.025277 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.026759 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.029309 (+ 2550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.029567 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.029641 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.029819 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12984,"thread_start_us":16311,"threads_started":1}
I20260927 22:43:37.034705 14256 tablet_service.cc:1467] Processing CreateTablet for tablet 94aab7b1e74a4b2c8804761cdc6af463 (DEFAULT_TABLE table=test-table [id=89ef6c41823d4dedb3c09c05b8f02a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:37.036581 14256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94aab7b1e74a4b2c8804761cdc6af463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:37.043540 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.013011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:46467 (local address 127.0.0.1:52800)
0927 22:43:37.022025 (+ 9014us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.022045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.022156 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.025557 (+ 3401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.025565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.025602 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.026092 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.026108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.036412 (+ 10304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.036432 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.039999 (+ 3567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.040019 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.042618 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.042626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.042811 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.042854 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.042943 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.043111 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8879,"thread_start_us":8739,"threads_started":1}
I20260927 22:43:37.044786 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.013278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.1.53.66:46467)
0927 22:43:37.024442 (+ 11164us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.024455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.024494 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:43:37.024617 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.024629 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.024988 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.025236 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.028669 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.030144 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.040308 (+ 10164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.042342 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.043967 (+ 1625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.044164 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.044261 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.044451 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10928,"thread_start_us":10780,"threads_started":1}
I20260927 22:43:37.048760 14400 tablet_service.cc:1467] Processing CreateTablet for tablet 94aab7b1e74a4b2c8804761cdc6af463 (DEFAULT_TABLE table=test-table [id=89ef6c41823d4dedb3c09c05b8f02a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:37.050326 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94aab7b1e74a4b2c8804761cdc6af463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:37.052014 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:36.999902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37887 (local address 127.0.0.1:60106)
0927 22:43:37.011306 (+ 11404us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.011362 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.011538 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.028177 (+ 16639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.028182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.028209 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.028552 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.028564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.042930 (+ 14366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.042935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.044904 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.044924 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.051195 (+ 6271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.051203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.051393 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.051445 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.051521 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.051670 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11123,"thread_start_us":12654,"threads_started":1}
I20260927 22:43:37.053229 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.000636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.1.53.67:37887)
0927 22:43:37.027161 (+ 26525us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.027170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.027206 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:43:37.027307 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.027317 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.027582 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.027782 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.030630 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.042606 (+ 11976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.045752 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.050927 (+ 5175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.052389 (+ 1462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.052647 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.052724 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.052897 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26343,"thread_start_us":27390,"threads_started":1}
I20260927 22:43:37.057145 14534 tablet_service.cc:1467] Processing CreateTablet for tablet 94aab7b1e74a4b2c8804761cdc6af463 (DEFAULT_TABLE table=test-table [id=89ef6c41823d4dedb3c09c05b8f02a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:37.058780 14534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94aab7b1e74a4b2c8804761cdc6af463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:37.072822 14620 tablet_bootstrap.cc:492] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Bootstrap starting.
I20260927 22:43:37.078788 14620 tablet_bootstrap.cc:654] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:37.080286 14620 log.cc:824] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:37.084966 14624 tablet_bootstrap.cc:492] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: Bootstrap starting.
I20260927 22:43:37.089399 14624 tablet_bootstrap.cc:654] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:37.094560 14620 tablet_bootstrap.cc:492] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: No bootstrap required, opened a new log
I20260927 22:43:37.094982 14620 ts_tablet_manager.cc:1397] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20260927 22:43:37.106192 14624 log.cc:824] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:37.109409 14620 raft_consensus.cc:348] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.111138 14620 raft_consensus.cc:374] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:37.111769 14620 raft_consensus.cc:729] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851b3026588841b68de9ec3b9168150b, State: Initialized, Role: FOLLOWER
I20260927 22:43:37.115046 14622 tablet_bootstrap.cc:492] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Bootstrap starting.
I20260927 22:43:37.116156 14620 consensus_queue.cc:260] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.118000 14626 raft_consensus.cc:482] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:37.119315 14624 tablet_bootstrap.cc:492] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: No bootstrap required, opened a new log
I20260927 22:43:37.118235 14626 raft_consensus.cc:504] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.119683 14624 ts_tablet_manager.cc:1397] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.006s
I20260927 22:43:37.129038 14622 tablet_bootstrap.cc:654] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:37.130484 14620 ts_tablet_manager.cc:1428] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Time spent starting tablet: real 0.035s user 0.011s sys 0.004s
I20260927 22:43:37.130841 14465 heartbeater.cc:503] Master 127.1.53.126:33919 was elected leader, sending a full tablet report...
I20260927 22:43:37.131050 14626 leader_election.cc:288] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887), 6305f305ab0843e1a0de6af543c6cbce (127.1.53.65:32827)
I20260927 22:43:37.132690 14622 log.cc:824] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:43:37 dist-test-slave-pp80 krb5kdc[13113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for HTTP/127.1.53.84@KRBTEST.COM
I20260927 22:43:37.151582 14624 raft_consensus.cc:348] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.157743 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.122529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:55877 (local address 127.1.53.67:37887)
0927 22:43:37.122755 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.122762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.128913 (+ 6151us) server_negotiation.cc:407] Connection header received
0927 22:43:37.129699 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.129705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.129775 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.129833 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.144717 (+ 14884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.145880 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.153133 (+ 7253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.154375 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.154481 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.154580 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.157314 (+ 2734us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.157478 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:43:37.158715 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.121795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37887 (local address 127.1.53.66:55877)
0927 22:43:37.128785 (+ 6990us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.128808 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.129525 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.130029 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.130033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.130058 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.130506 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.130519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.146137 (+ 15618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.146309 (+ 172us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.149491 (+ 3182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.149514 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.155861 (+ 6347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.155868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.156055 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.156091 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.158192 (+ 2101us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.158354 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6786,"thread_start_us":6617,"threads_started":1}
I20260927 22:43:37.162688 14624 raft_consensus.cc:374] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:37.163761 14624 raft_consensus.cc:729] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6305f305ab0843e1a0de6af543c6cbce, State: Initialized, Role: FOLLOWER
I20260927 22:43:37.164927 14624 consensus_queue.cc:260] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.168648 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.133021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:41955 (local address 127.1.53.65:32827)
0927 22:43:37.133199 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.133205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.152833 (+ 19628us) server_negotiation.cc:407] Connection header received
0927 22:43:37.156318 (+ 3485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.156325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.156439 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.156509 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.159045 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.160129 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.165729 (+ 5600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.167179 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.167321 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.167458 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.168160 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.168362 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:43:37.165432 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94aab7b1e74a4b2c8804761cdc6af463" candidate_uuid: "851b3026588841b68de9ec3b9168150b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62c96ba453644ea85ceccfe90db36a9" is_pre_election: true
I20260927 22:43:37.169688 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.135837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.1.53.66:41955)
0927 22:43:37.152716 (+ 16879us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.152744 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.155578 (+ 2834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.156708 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.156713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.156743 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.157110 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.157121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.160382 (+ 3261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.160387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.162410 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.162427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.167700 (+ 5273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.167709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.167933 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.167973 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.169108 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.169290 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16734,"thread_start_us":17016,"threads_started":1}
W20260927 22:43:37.170811 14354 leader_election.cc:341] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887): Illegal state: must be running to vote when last-logged opid is not known
I20260927 22:43:37.171651 14276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94aab7b1e74a4b2c8804761cdc6af463" candidate_uuid: "851b3026588841b68de9ec3b9168150b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6305f305ab0843e1a0de6af543c6cbce" is_pre_election: true
I20260927 22:43:37.172374 14276 raft_consensus.cc:2454] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851b3026588841b68de9ec3b9168150b in term 0.
I20260927 22:43:37.173240 14355 leader_election.cc:302] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6305f305ab0843e1a0de6af543c6cbce, 851b3026588841b68de9ec3b9168150b; no voters: e62c96ba453644ea85ceccfe90db36a9
I20260927 22:43:37.173805 14627 raft_consensus.cc:2781] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:43:37.173988 14627 raft_consensus.cc:482] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:43:37.174120 14627 raft_consensus.cc:3037] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:37.177023 14321 heartbeater.cc:503] Master 127.1.53.126:33919 was elected leader, sending a full tablet report...
I20260927 22:43:37.177803 14627 raft_consensus.cc:504] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.179078 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94aab7b1e74a4b2c8804761cdc6af463" candidate_uuid: "851b3026588841b68de9ec3b9168150b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62c96ba453644ea85ceccfe90db36a9"
W20260927 22:43:37.179796 14354 leader_election.cc:341] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887): Illegal state: must be running to vote when last-logged opid is not known
I20260927 22:43:37.180722 14276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94aab7b1e74a4b2c8804761cdc6af463" candidate_uuid: "851b3026588841b68de9ec3b9168150b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6305f305ab0843e1a0de6af543c6cbce"
I20260927 22:43:37.180964 14276 raft_consensus.cc:3037] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:37.184415 14276 raft_consensus.cc:2454] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851b3026588841b68de9ec3b9168150b in term 1.
I20260927 22:43:37.184916 14355 leader_election.cc:302] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6305f305ab0843e1a0de6af543c6cbce, 851b3026588841b68de9ec3b9168150b; no voters: e62c96ba453644ea85ceccfe90db36a9
I20260927 22:43:37.185259 14626 raft_consensus.cc:2781] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:37.185325 14627 leader_election.cc:288] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [CANDIDATE]: Term 1 election: Requested vote from peers e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887), 6305f305ab0843e1a0de6af543c6cbce (127.1.53.65:32827)
I20260927 22:43:37.188570 14624 ts_tablet_manager.cc:1428] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: Time spent starting tablet: real 0.069s user 0.011s sys 0.005s
I20260927 22:43:37.189006 14626 raft_consensus.cc:686] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 1 LEADER]: Becoming Leader. State: Replica: 851b3026588841b68de9ec3b9168150b, State: Running, Role: LEADER
I20260927 22:43:37.189291 14622 tablet_bootstrap.cc:492] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: No bootstrap required, opened a new log
I20260927 22:43:37.189633 14622 ts_tablet_manager.cc:1397] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.027s
I20260927 22:43:37.189641 14626 consensus_queue.cc:237] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.196086 14123 catalog_manager.cc:5495] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b reported cstate change: term changed from 0 to 1, leader changed from <none> to 851b3026588841b68de9ec3b9168150b (127.1.53.66). New cstate: current_term: 1 leader_uuid: "851b3026588841b68de9ec3b9168150b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } health_report { overall_health: HEALTHY } } }
I20260927 22:43:37.197389 14622 raft_consensus.cc:348] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.197835 14622 raft_consensus.cc:374] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:37.197949 14622 raft_consensus.cc:729] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62c96ba453644ea85ceccfe90db36a9, State: Initialized, Role: FOLLOWER
I20260927 22:43:37.198644 14622 consensus_queue.cc:260] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.202019 14599 heartbeater.cc:503] Master 127.1.53.126:33919 was elected leader, sending a full tablet report...
I20260927 22:43:37.202785 14622 ts_tablet_manager.cc:1428] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
W20260927 22:43:37.239054 14466 tablet.cc:2369] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:37.255400 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.247394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.1.53.66:46467)
0927 22:43:37.247576 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.247583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.247629 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:43:37.247925 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.247930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.248052 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.248226 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:43:37.248904 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.250030 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.252066 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.252693 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.252819 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.253555 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:43:37.254346 (+ 791us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:43:37.254513 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.255016 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.255104 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:43:37.260923 14634 raft_consensus.cc:482] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:37.261233 14634 raft_consensus.cc:504] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 } } peers { permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 } } peers { permanent_uuid: "851b3026588841b68de9ec3b9168150b" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 46467 } }
I20260927 22:43:37.263053 14634 leader_election.cc:288] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6305f305ab0843e1a0de6af543c6cbce (127.1.53.65:32827), 851b3026588841b68de9ec3b9168150b (127.1.53.66:46467)
I20260927 22:43:37.272564 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.263605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:46467 (local address 127.1.53.67:59163)
0927 22:43:37.263912 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.263931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.264043 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.266485 (+ 2442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.266490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.266529 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.266719 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.266730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.268214 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.268219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.270192 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.270207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.271821 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.271828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.272018 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.272055 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.272127 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.272280 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260927 22:43:37.273722 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.264506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:59163 (local address 127.1.53.66:46467)
0927 22:43:37.266154 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.266160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.266180 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:43:37.266247 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.266253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.266324 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.266849 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.266989 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.268059 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.270448 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.271634 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.273175 (+ 1541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.273276 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.273327 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.273461 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517}
I20260927 22:43:37.274775 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94aab7b1e74a4b2c8804761cdc6af463" candidate_uuid: "e62c96ba453644ea85ceccfe90db36a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851b3026588841b68de9ec3b9168150b" is_pre_election: true
I20260927 22:43:37.277734 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.265368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:58197 (local address 127.1.53.65:32827)
0927 22:43:37.265535 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.265541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.266260 (+ 719us) server_negotiation.cc:407] Connection header received
0927 22:43:37.267201 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.267208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.267302 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.267368 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:37.268762 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.269861 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.274640 (+ 4779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.275938 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.276046 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.276151 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.277313 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.277454 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:43:37.278424 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.264944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.1.53.67:58197)
0927 22:43:37.266149 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.266169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:37.266852 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:37.267589 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:37.267594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:37.267618 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:37.267971 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.267983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.270098 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.270103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.274290 (+ 4187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:37.274305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.276444 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.276450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:37.276622 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.276656 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:43:37.277946 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0927 22:43:37.278094 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":900,"threads_started":1}
I20260927 22:43:37.279539 14276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94aab7b1e74a4b2c8804761cdc6af463" candidate_uuid: "e62c96ba453644ea85ceccfe90db36a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6305f305ab0843e1a0de6af543c6cbce" is_pre_election: true
I20260927 22:43:37.279892 14276 raft_consensus.cc:2379] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e62c96ba453644ea85ceccfe90db36a9 in current term 1: Already voted for candidate 851b3026588841b68de9ec3b9168150b in this term.
I20260927 22:43:37.280745 14490 leader_election.cc:302] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e62c96ba453644ea85ceccfe90db36a9; no voters: 6305f305ab0843e1a0de6af543c6cbce, 851b3026588841b68de9ec3b9168150b
I20260927 22:43:37.281281 14634 raft_consensus.cc:3037] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:37.284462 14634 raft_consensus.cc:2726] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 22:43:37.285300 14322 tablet.cc:2369] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:37.291659 14276 raft_consensus.cc:1261] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce [term 1 FOLLOWER]: Refusing update from remote peer 851b3026588841b68de9ec3b9168150b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:37.293342 14626 consensus_queue.cc:1035] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6305f305ab0843e1a0de6af543c6cbce" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:37.300693 14554 raft_consensus.cc:1261] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 1 FOLLOWER]: Refusing update from remote peer 851b3026588841b68de9ec3b9168150b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:37.301731 14626 consensus_queue.cc:1035] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62c96ba453644ea85ceccfe90db36a9" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:37.332288 14636 mvcc.cc:205] Tried to move back new op lower bound from 7334088774812893184 to 7334088774416998400. Current Snapshot: MvccSnapshot[applied={T|T < 7334088774812893184}]
I20260927 22:43:37.334987 14638 mvcc.cc:205] Tried to move back new op lower bound from 7334088774812893184 to 7334088774416998400. Current Snapshot: MvccSnapshot[applied={T|T < 7334088774812893184}]
I20260927 22:43:37.357316 14639 mvcc.cc:205] Tried to move back new op lower bound from 7334088774812893184 to 7334088774416998400. Current Snapshot: MvccSnapshot[applied={T|T < 7334088774812893184}]
I20260927 22:43:37.374523 14123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:43:37.374873 14123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:43:37.379597 14123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6e51a154f64dc8a2d63945040c3b94: Sending DeleteTablet for 3 replicas of tablet 94aab7b1e74a4b2c8804761cdc6af463
I20260927 22:43:37.380759 14534 tablet_service.cc:1514] Processing DeleteTablet for tablet 94aab7b1e74a4b2c8804761cdc6af463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:37 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:60106
I20260927 22:43:37.382253 14256 tablet_service.cc:1514] Processing DeleteTablet for tablet 94aab7b1e74a4b2c8804761cdc6af463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:37 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:56954
I20260927 22:43:37.388942 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet 94aab7b1e74a4b2c8804761cdc6af463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:37 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:52800
I20260927 22:43:37.397189 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14161
I20260927 22:43:37.430888 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.398686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.1.53.65:32827)
0927 22:43:37.398872 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.398879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.398901 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:43:37.399068 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.399076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.399172 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.399271 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:37.402318 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.403542 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.405457 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.406246 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.406399 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.422626 (+ 16227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:37.422654 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:37.422681 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:37.422765 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:37.427378 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:37.428415 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:37.428430 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:37.428437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:37.428639 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:37.429044 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:37.429051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:37.429056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:37.429603 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:37.429905 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.430237 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.430469 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:43:37.431746 14316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:37.436774 14648 tablet_replica.cc:330] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: stopping tablet replica
I20260927 22:43:37.437213 14648 raft_consensus.cc:2229] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:43:37.437849 14648 raft_consensus.cc:2258] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:37.443537 14646 tablet_replica.cc:330] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: stopping tablet replica
I20260927 22:43:37.444114 14646 raft_consensus.cc:2229] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:43:37.444865 14646 raft_consensus.cc:2258] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:37.444999 14648 ts_tablet_manager.cc:1905] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:37.447168 14646 ts_tablet_manager.cc:1905] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:37.453117 14648 ts_tablet_manager.cc:1918] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:37.453414 14648 log.cc:1196] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/94aab7b1e74a4b2c8804761cdc6af463
I20260927 22:43:37.454047 14648 ts_tablet_manager.cc:1939] T 94aab7b1e74a4b2c8804761cdc6af463 P 851b3026588841b68de9ec3b9168150b: Deleting consensus metadata
I20260927 22:43:37.454607 14646 ts_tablet_manager.cc:1918] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:37.454789 14646 log.cc:1196] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/94aab7b1e74a4b2c8804761cdc6af463
I20260927 22:43:37.455296 14646 ts_tablet_manager.cc:1939] T 94aab7b1e74a4b2c8804761cdc6af463 P e62c96ba453644ea85ceccfe90db36a9: Deleting consensus metadata
I20260927 22:43:37.456215 14110 catalog_manager.cc:4841] TS 851b3026588841b68de9ec3b9168150b (127.1.53.66:46467): tablet 94aab7b1e74a4b2c8804761cdc6af463 (table test-table [id=89ef6c41823d4dedb3c09c05b8f02a2c]) successfully deleted
I20260927 22:43:37.460484 14110 catalog_manager.cc:4841] TS e62c96ba453644ea85ceccfe90db36a9 (127.1.53.67:37887): tablet 94aab7b1e74a4b2c8804761cdc6af463 (table test-table [id=89ef6c41823d4dedb3c09c05b8f02a2c]) successfully deleted
I20260927 22:43:37.535233 14647 tablet_replica.cc:330] T 94aab7b1e74a4b2c8804761cdc6af463 P 6305f305ab0843e1a0de6af543c6cbce: stopping tablet replica
I20260927 22:43:37.535570 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14161
W20260927 22:43:37.552999 14111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:32827 (error 108)
I20260927 22:43:37.553200 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14325
I20260927 22:43:37.555186 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.554398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:56974)
0927 22:43:37.554865 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.554987 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
W20260927 22:43:37.555753 14111 catalog_manager.cc:4584] TS 6305f305ab0843e1a0de6af543c6cbce (127.1.53.65:32827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94aab7b1e74a4b2c8804761cdc6af463: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
I20260927 22:43:37.570533 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.554669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.1.53.66:46467)
0927 22:43:37.554857 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.554864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.554889 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:43:37.554955 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.554960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.555084 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.555193 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:37.556735 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.557876 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.559182 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.559851 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.559983 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.562812 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:37.562830 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:37.562850 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:37.562896 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:37.567182 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:37.567939 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:37.567949 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:37.567953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:37.568132 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:37.568745 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:37.568751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:37.568755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:37.569245 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:37.569594 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.569930 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.570164 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:43:37.571445 14460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:37.609529 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.609042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:56984)
0927 22:43:37.609272 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.609374 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:43:37.610565 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.610207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:56988)
0927 22:43:37.610357 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.610431 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:43:37.671103 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.670574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:56992)
0927 22:43:37.670849 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.670940 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 22:43:37.672438 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.671935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:57008)
0927 22:43:37.672132 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.672232 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:43:37.687634 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14325
I20260927 22:43:37.708350 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14469
I20260927 22:43:37.722422 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.709222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.1.53.67:37887)
0927 22:43:37.709744 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:37.709753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:37.709776 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:43:37.709832 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:37.709836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:37.709911 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:37.710026 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:37.711356 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.712114 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.713528 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:37.713917 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:37.714063 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:37.716731 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:37.716745 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:37.716762 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:37.716797 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:37.719571 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:37.720297 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:37.720313 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:37.720315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:37.720450 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:37.720845 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:37.720857 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:37.720860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:37.721216 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:37.721522 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:37.721811 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:43:37.722043 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":261,"threads_started":1}
I20260927 22:43:37.723311 14594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:37.753296 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.752722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:57016)
0927 22:43:37.752959 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.753075 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:43:37.754886 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.754352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:57026)
0927 22:43:37.754601 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.754693 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 22:43:37.819259 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14469
I20260927 22:43:37.836105 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14091
I20260927 22:43:37.837477 14154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:37.959064 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:37.958244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:32827 (local address 127.0.0.1:57034)
0927 22:43:37.958645 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:37.958808 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:43:37.959600 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14091
I20260927 22:43:37.983886 1237 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:43:38.323086 1237 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:43:38.323333 1237 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:43:38.668893 1237 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:43:38.669 UTC [13143] LOG: received smart shutdown request
2026-09-27 22:43:38.672 UTC [13143] LOG: background worker "logical replication launcher" (PID 13152) exited with exit code 1
2026-09-27 22:43:38.673 UTC [13147] LOG: shutting down
2026-09-27 22:43:38.870 UTC [13143] LOG: database system is shut down
2026-09-27T22:43:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54246 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:43:39 dist-test-slave-pp80 krb5kdc[14671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:43:39 dist-test-slave-pp80 krb5kdc[14671](info): set up 2 sockets
Sep 27 22:43:39 dist-test-slave-pp80 krb5kdc[14671](info): commencing operation
krb5kdc: starting...
W20260927 22:43:40.116616 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:43:40 dist-test-slave-pp80 krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:43:40.360765 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37717
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37717,127.1.53.125:46519,127.1.53.124:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:40.707980 14684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:40.708379 14684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:40.708554 14684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:40.716032 14684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:40.716187 14684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:40.716310 14684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:40.716414 14684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:40.716491 14684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:40.726666 14684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37717,127.1.53.125:46519,127.1.53.124:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:40.728109 14684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:40.729480 14684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:40.741482 14689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:40.741587 14690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:40.741515 14692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:40.745795 14684 server_base.cc:1034] running on GCE node
I20260927 22:43:40.746706 14684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:40.748205 14684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:40.749470 14684 hybrid_clock.cc:648] HybridClock initialized: now 1790549020749414 us; error 55 us; skew 500 ppm
I20260927 22:43:40.749847 14684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:40 dist-test-slave-pp80 krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:40.757617 14684 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:40.759359 14684 webserver.cc:460] Webserver started at http://127.1.53.126:32957/ using document root <none> and password file <none>
I20260927 22:43:40.760226 14684 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:40.760528 14684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:40.760913 14684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:40.764119 14684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "13de2c3352304577ab16e0d3852366b5"
format_stamp: "Formatted at 2026-09-27 22:43:40 on dist-test-slave-pp80"
server_key: "459beabf0d04c2ec7119b4792e5eebce"
server_key_iv: "3b76f03e5b0f5a5eb0cd8c07be2cd9ed"
server_key_version: "encryptionkey@0"
I20260927 22:43:40.765069 14684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "13de2c3352304577ab16e0d3852366b5"
format_stamp: "Formatted at 2026-09-27 22:43:40 on dist-test-slave-pp80"
server_key: "459beabf0d04c2ec7119b4792e5eebce"
server_key_iv: "3b76f03e5b0f5a5eb0cd8c07be2cd9ed"
server_key_version: "encryptionkey@0"
I20260927 22:43:40.770587 14684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:43:40.774232 14699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:40.775928 14684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:43:40.776166 14684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "13de2c3352304577ab16e0d3852366b5"
format_stamp: "Formatted at 2026-09-27 22:43:40 on dist-test-slave-pp80"
server_key: "459beabf0d04c2ec7119b4792e5eebce"
server_key_iv: "3b76f03e5b0f5a5eb0cd8c07be2cd9ed"
server_key_version: "encryptionkey@0"
I20260927 22:43:40.776407 14684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:40.798924 14684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:40.803318 14684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:40.803629 14684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:40.823171 14684 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37717
I20260927 22:43:40.823205 14750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37717 every 8 connection(s)
I20260927 22:43:40.824656 14684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:40.825436 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14684
I20260927 22:43:40.825691 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:43:40.826393 1237 external_mini_cluster.cc:1428] Setting key 6fb1c095272ee8c65b339e530474c1e4
I20260927 22:43:40.829399 14751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:40.836153 14751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.125@KRBTEST.COM" created.
Sep 27 22:43:40 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:40.859419 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.839632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37717 (local address 127.0.0.1:47246)
0927 22:43:40.840883 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.840922 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:40.842731 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:40.843574 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:40.843581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:40.844355 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:40.844697 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:40.844715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:40.846326 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:40.846331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:40.847392 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:40.847403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:40.848402 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:40.848409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:40.848583 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:40.849269 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:40.849292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:40.851670 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:40.854989 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:40.855000 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:40.855013 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:40.855390 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:40.855800 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:40.855807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:40.855810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:40.855972 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:40.856874 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:40.856882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:40.857100 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 22:43:40.857345 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 22:43:40.858007 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":197,"thread_start_us":834,"threads_started":1}
I20260927 22:43:40.859462 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.839632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.1.53.126:37717)
0927 22:43:40.840624 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:40.840640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:40.841063 (+ 423us) server_negotiation.cc:407] Connection header received
0927 22:43:40.843114 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:40.843120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:40.843345 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:40.843561 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:40.844894 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:40.846174 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:40.847540 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:40.848154 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:40.848375 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:40.851900 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:40.851909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:40.851922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:40.851960 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:40.854827 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:40.855536 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:40.855543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:40.855545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:40.855690 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:40.856086 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:40.856092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:40.856100 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:40.856716 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:40.856927 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:40.857486 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 22:43:40.857889 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":719,"thread_start_us":462,"threads_started":1}
I20260927 22:43:40.862655 14751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:43:40.863904 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.863365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44864)
0927 22:43:40.863578 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.863727 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 22:43:40.864228 14700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
I20260927 22:43:40.866024 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.865537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44874)
0927 22:43:40.865725 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.865819 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:43:40.866407 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:46519: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab.
Entry for principal kudu/127.1.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab.
I20260927 22:43:40.876305 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:46519
--webserver_interface=127.1.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab
--principal=kudu/127.1.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37717,127.1.53.125:46519,127.1.53.124:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:40.915867 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } attempt: 1
I20260927 22:43:40.917174 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.916668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44878)
0927 22:43:40.916871 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.916990 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:43:40.918375 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.917967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44880)
0927 22:43:40.918180 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.918251 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 22:43:40.918665 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:46519: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
I20260927 22:43:40.986958 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } attempt: 2
I20260927 22:43:40.988210 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.987726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44884)
0927 22:43:40.987954 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.988061 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:43:40.989384 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:40.989007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44886)
0927 22:43:40.989158 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:40.989251 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:43:40.989565 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:46519: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
I20260927 22:43:41.080802 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } attempt: 3
I20260927 22:43:41.081983 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.081486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44894)
0927 22:43:41.081684 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.081793 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:43:41.083328 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.082876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44904)
0927 22:43:41.083068 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.083159 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:43:41.083618 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:46519: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
W20260927 22:43:41.224897 14758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:41.225207 14758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:41.225374 14758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:43:41.226876 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } attempt: 4
I20260927 22:43:41.228138 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.227536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44918)
0927 22:43:41.227866 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.227977 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260927 22:43:41.229260 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.228838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44922)
0927 22:43:41.228993 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.229103 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:43:41.229512 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:46519: Network error: Client connection negotiation failed: client connection to 127.1.53.125:46519: connect: Connection refused (error 111)
W20260927 22:43:41.233652 14758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:41.233762 14758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:41.233834 14758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:41.233947 14758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:41.234032 14758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:41.244172 14758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37717,127.1.53.125:46519,127.1.53.124:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab
--principal=kudu/127.1.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:41.245606 14758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:41.246958 14758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:41.259076 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:41.259076 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:41.259321 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:41.260345 14758 server_base.cc:1034] running on GCE node
I20260927 22:43:41.261215 14758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:41.262701 14758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:41.263938 14758 hybrid_clock.cc:648] HybridClock initialized: now 1790549021263888 us; error 44 us; skew 500 ppm
I20260927 22:43:41.264394 14758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:41.267800 14758 init.cc:377] Logged in from keytab as kudu/127.1.53.125@KRBTEST.COM (short username kudu)
I20260927 22:43:41.269446 14758 webserver.cc:460] Webserver started at http://127.1.53.125:38689/ using document root <none> and password file <none>
I20260927 22:43:41.270167 14758 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:41.270320 14758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:41.270634 14758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:41.273758 14758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data/instance:
uuid: "87b73f0c472444518bd292ff69983648"
format_stamp: "Formatted at 2026-09-27 22:43:41 on dist-test-slave-pp80"
server_key: "07a20e1f375a530ffe87e2db966b6867"
server_key_iv: "24216886e1ec1b2ba011712b8aabd15f"
server_key_version: "encryptionkey@0"
I20260927 22:43:41.274628 14758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal/instance:
uuid: "87b73f0c472444518bd292ff69983648"
format_stamp: "Formatted at 2026-09-27 22:43:41 on dist-test-slave-pp80"
server_key: "07a20e1f375a530ffe87e2db966b6867"
server_key_iv: "24216886e1ec1b2ba011712b8aabd15f"
server_key_version: "encryptionkey@0"
I20260927 22:43:41.280769 14758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 22:43:41.284556 14773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:41.285984 14758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:43:41.286181 14758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal
uuid: "87b73f0c472444518bd292ff69983648"
format_stamp: "Formatted at 2026-09-27 22:43:41 on dist-test-slave-pp80"
server_key: "07a20e1f375a530ffe87e2db966b6867"
server_key_iv: "24216886e1ec1b2ba011712b8aabd15f"
server_key_version: "encryptionkey@0"
I20260927 22:43:41.286384 14758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:41.315912 14758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:41.320833 14758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:41.321158 14758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:41.341665 14758 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.125:46519
I20260927 22:43:41.341687 14824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.125:46519 every 8 connection(s)
I20260927 22:43:41.343110 14758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
I20260927 22:43:41.347064 14825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:41.349201 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14758
I20260927 22:43:41.349416 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-1/wal/instance
I20260927 22:43:41.349891 1237 external_mini_cluster.cc:1428] Setting key 2d8824351d707925d4adc8f1bc41424d
I20260927 22:43:41.353857 14825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.124@KRBTEST.COM" created.
I20260927 22:43:41.376911 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.356909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.1.53.126:37717)
0927 22:43:41.357111 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.357116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.358476 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 22:43:41.360470 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.360479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.360580 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.360716 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.362352 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.363509 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.365222 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.366032 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.366176 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.370038 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.370043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.370046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.370081 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.373349 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.374142 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.374151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.374158 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.374305 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.374760 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.374767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.374774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.375198 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.375353 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.376165 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.376463 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:43:41.377617 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.357407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37717 (local address 127.0.0.1:47262)
0927 22:43:41.358271 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.358319 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.360101 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.361010 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.361018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.361557 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.362158 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.362177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.363693 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.363698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.365070 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.365083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.366270 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.366276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.366467 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.367181 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.367204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.369743 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.373512 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.373519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.373532 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.373967 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.374443 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.374447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.374449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.374631 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.375355 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.375361 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.375702 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.375998 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.376479 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":387,"threads_started":1}
I20260927 22:43:41.379573 14825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:43:41.398020 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.380853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44936)
0927 22:43:41.381162 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.381187 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.381303 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.382243 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.382254 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.382796 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.383021 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.383036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.384505 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.384513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.386203 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.386220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.387097 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.387106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.387345 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.388000 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.388027 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.391231 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.395570 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.395574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.395577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.395935 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.396426 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.396430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.396432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.396583 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.397298 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.397301 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.397411 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.397530 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.397736 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:43:41.398365 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.381093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.1.53.125:46519)
0927 22:43:41.381791 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.381797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.381825 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:43:41.381873 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.381879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.382076 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.382201 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.383193 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.384374 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.386373 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.386930 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.387039 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.391518 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.391535 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.391556 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.391600 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.395361 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.396082 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.396089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.396091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.396321 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.396683 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.396686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.396688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.397123 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.397357 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.397713 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.397914 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":404,"threads_started":1}
I20260927 22:43:41.400972 14825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:43:41.402079 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.401581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54472)
0927 22:43:41.401745 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.401940 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:43:41.402393 14777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab.
Entry for principal kudu/127.1.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab.
I20260927 22:43:41.404338 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.403815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54486)
0927 22:43:41.404061 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.404150 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:43:41.404580 14825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.405071 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:35929
--webserver_interface=127.1.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab
--principal=kudu/127.1.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37717,127.1.53.125:46519,127.1.53.124:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:41.453904 14825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 1
I20260927 22:43:41.455257 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.454583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54492)
0927 22:43:41.454853 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.455029 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:43:41.456410 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.455991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54506)
0927 22:43:41.456149 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.456279 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:43:41.456650 14825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.524943 14825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 2
I20260927 22:43:41.526276 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.525672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54516)
0927 22:43:41.525964 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.526088 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:43:41.527395 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.527020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54532)
0927 22:43:41.527167 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.527253 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:43:41.527668 14825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.528789 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } attempt: 5
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:43:41.542953 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.529458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.1.53.125:46519)
0927 22:43:41.529612 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.529618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.529683 (+ 65us) server_negotiation.cc:407] Connection header received
0927 22:43:41.529841 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.529847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.529952 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.530067 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.530839 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.531923 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.533191 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.533880 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.534010 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.537571 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.537576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.537579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.537612 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.540325 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.541134 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.541139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.541141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.541258 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.541708 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.541711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.541714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.542012 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.542146 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.542565 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.542733 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:43:41.543110 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.529412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44942)
0927 22:43:41.529584 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.529610 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.529723 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.530109 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.530113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.530520 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.530693 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.530702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.532077 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.532081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.533046 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.533059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.534034 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.534039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.534167 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.534881 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.534906 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.537365 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.540493 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.540498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.540502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.541002 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.541372 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.541377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.541381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.541576 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.542146 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.542151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.542330 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.542484 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.542764 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:43:41.544376 14751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:43:41.545595 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.545042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54542)
0927 22:43:41.545276 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.545392 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:43:41.546794 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.546396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54554)
0927 22:43:41.546572 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.546672 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:43:41.547045 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.598312 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 1
I20260927 22:43:41.599720 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.599068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54568)
0927 22:43:41.599335 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.599479 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:43:41.600981 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.600512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54570)
0927 22:43:41.600702 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.600804 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:43:41.601269 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.618986 14825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 3
I20260927 22:43:41.620304 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.619676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54584)
0927 22:43:41.619941 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.620059 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:43:41.621644 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.621141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54600)
0927 22:43:41.621390 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.621486 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 22:43:41.621917 14825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.669529 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 2
I20260927 22:43:41.670893 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.670268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54604)
0927 22:43:41.670522 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.670677 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:43:41.672021 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.671606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54620)
0927 22:43:41.671779 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.671868 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:43:41.672317 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
W20260927 22:43:41.743422 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:41.743707 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:41.743871 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:41.751232 14832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:41.751345 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:41.751448 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:41.751545 14832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:41.751623 14832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:41.761889 14832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37717,127.1.53.125:46519,127.1.53.124:35929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab
--principal=kudu/127.1.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:41.763301 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:41.764767 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:43:41.765163 14825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 4
I20260927 22:43:41.766446 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.765887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54626)
0927 22:43:41.766147 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.766256 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:43:41.767652 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.767257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54632)
0927 22:43:41.767413 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.767502 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:43:41.767941 14825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.778568 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 3
W20260927 22:43:41.778792 14838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:41.778771 14837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:41.779768 14840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:41.780114 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.779422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54634)
0927 22:43:41.779670 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.779819 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:43:41.781332 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.780891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54650)
0927 22:43:41.781125 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.781212 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 22:43:41.781584 14751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:35929: Network error: Client connection negotiation failed: client connection to 127.1.53.124:35929: connect: Connection refused (error 111)
I20260927 22:43:41.786355 14832 server_base.cc:1034] running on GCE node
I20260927 22:43:41.787263 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:41.788895 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:41.790153 14832 hybrid_clock.cc:648] HybridClock initialized: now 1790549021790084 us; error 70 us; skew 500 ppm
I20260927 22:43:41.790514 14832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:41.793843 14832 init.cc:377] Logged in from keytab as kudu/127.1.53.124@KRBTEST.COM (short username kudu)
I20260927 22:43:41.795655 14832 webserver.cc:460] Webserver started at http://127.1.53.124:41921/ using document root <none> and password file <none>
I20260927 22:43:41.796429 14832 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:41.796538 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:41.796808 14832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:41.799821 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data/instance:
uuid: "8b3a8442c2c94bf9b2af1a3243038f3b"
format_stamp: "Formatted at 2026-09-27 22:43:41 on dist-test-slave-pp80"
server_key: "a8f3df683d4e3a8df6f28eb3b397cfd9"
server_key_iv: "252b467636472bec82855e81f99db911"
server_key_version: "encryptionkey@0"
I20260927 22:43:41.800690 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal/instance:
uuid: "8b3a8442c2c94bf9b2af1a3243038f3b"
format_stamp: "Formatted at 2026-09-27 22:43:41 on dist-test-slave-pp80"
server_key: "a8f3df683d4e3a8df6f28eb3b397cfd9"
server_key_iv: "252b467636472bec82855e81f99db911"
server_key_version: "encryptionkey@0"
I20260927 22:43:41.806064 14832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:43:41.810156 14847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:41.811753 14832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:43:41.811960 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal
uuid: "8b3a8442c2c94bf9b2af1a3243038f3b"
format_stamp: "Formatted at 2026-09-27 22:43:41 on dist-test-slave-pp80"
server_key: "a8f3df683d4e3a8df6f28eb3b397cfd9"
server_key_iv: "252b467636472bec82855e81f99db911"
server_key_version: "encryptionkey@0"
I20260927 22:43:41.812186 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:41.831889 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:41.837110 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:41.837447 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:41.857749 14832 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.124:35929
I20260927 22:43:41.857800 14898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.124:35929 every 8 connection(s)
I20260927 22:43:41.859290 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
I20260927 22:43:41.863392 14899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:41.867868 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14832
I20260927 22:43:41.868163 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790548812741971-1237-0/minicluster-data/master-2/wal/instance
I20260927 22:43:41.868685 1237 external_mini_cluster.cc:1428] Setting key 82d9f542176410a7dcd8a49999bde5f3
I20260927 22:43:41.870904 14899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:43:41.890856 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.873699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.1.53.126:37717)
0927 22:43:41.873932 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.873939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.874936 (+ 997us) server_negotiation.cc:407] Connection header received
0927 22:43:41.875234 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.875240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.875316 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.875441 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.876898 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.877602 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.879048 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.879523 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.879692 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.883461 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.883470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.883477 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.883523 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.887298 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.888162 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.888169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.888173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.888392 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.888858 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.888865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.888868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.889159 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.889278 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.890253 (+ 975us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.890487 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:43:41.892304 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.873738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37717 (local address 127.0.0.1:47276)
0927 22:43:41.874783 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.874834 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.875067 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.875502 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.875509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.876134 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.876729 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.876744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.877762 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.877772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.878901 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.878912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.879760 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.879771 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.879950 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.880717 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.880753 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.883218 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.887479 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.887490 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.887500 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.888028 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.888521 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.888525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.888528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.888732 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.889295 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.889306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.889787 (+ 481us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.890159 (+ 372us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.890725 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":3,"thread_start_us":549,"threads_started":1}
I20260927 22:43:41.892625 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.870787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.1.53.124:35929)
0927 22:43:41.871703 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.871718 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.871760 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:43:41.873698 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.873727 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.874186 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.874611 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.876140 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.877581 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.878983 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.879525 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.879722 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.885233 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.885245 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.885263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.885310 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.888746 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.889726 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.889734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.889737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.889953 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.890477 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.890484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.890488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.890967 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.891269 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.891790 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.892021 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":699,"thread_start_us":433,"threads_started":1}
I20260927 22:43:41.894155 14899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:41.910681 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.895149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:46519 (local address 127.0.0.1:44948)
0927 22:43:41.895327 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.895349 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.895475 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.895845 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.895850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.896347 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.896584 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.896596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.898149 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.898161 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.899544 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.899559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.900735 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.900739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.900913 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.901541 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.901560 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.904103 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.907923 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.907930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.907936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.908501 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.908975 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.908981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.908984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.909171 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.909737 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.909744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.909925 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.910089 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.910391 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:43:41.910781 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.895080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.1.53.125:46519)
0927 22:43:41.895255 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.895262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.895413 (+ 151us) server_negotiation.cc:407] Connection header received
0927 22:43:41.895601 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.895607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.895684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.895791 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.896750 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.897990 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.899712 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.900559 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.900695 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.904352 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.904362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.904370 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.904421 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.907779 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.908630 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.908637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.908643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.908833 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.909322 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.909326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.909329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.909606 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.909727 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.910175 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.910347 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:43:41.912199 14899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:43:41.913309 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.896129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.1.53.126:37717)
0927 22:43:41.896338 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.896343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.896361 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:43:41.896541 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.896546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.896616 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.896717 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.898363 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.899550 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.901337 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.901818 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.901920 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.906859 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.906864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.906867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.906901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.910305 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.911166 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.911173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.911178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.911347 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.911775 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.911781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.911787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.912143 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.912461 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.912768 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.912989 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:43:41.927263 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.913096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.1.53.124:35929)
0927 22:43:41.913268 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.913274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.913660 (+ 386us) server_negotiation.cc:407] Connection header received
0927 22:43:41.914226 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.914255 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.914327 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.914428 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.915313 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.916126 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.917696 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.918355 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.918443 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.922096 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.922101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.922104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.922131 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.924775 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.925359 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.925366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.925369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.925493 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.925858 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.925864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.925872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.926132 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.926276 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.926614 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.926884 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":66}
I20260927 22:43:41.927336 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.913147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54658)
0927 22:43:41.913588 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.913607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.913712 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.914463 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.914469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.914967 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.915167 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.915178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.916275 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.916281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.917560 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.917571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.918501 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.918507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.918726 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.919323 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.919344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.921902 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.924900 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.924908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.924912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.925245 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.925596 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.925603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.925608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.925738 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.926241 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.926245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.926386 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.926535 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.926986 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":163}
I20260927 22:43:41.930665 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.914963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.1.53.125:46519)
0927 22:43:41.915187 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.915192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.915209 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:43:41.915359 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.915367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.915445 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.915549 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.917200 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.918117 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.919860 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.920348 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.920443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.924916 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.924921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.924924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.924956 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.927890 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.928706 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.928713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.928717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.928854 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.929229 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.929233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.929235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.929494 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.929783 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.930116 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.930353 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260927 22:43:41.932701 14899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: Bootstrap starting.
I20260927 22:43:41.935678 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:43:41.936688 14899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:41.937867 14899 log.cc:824] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:41.941426 14899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: No bootstrap required, opened a new log
I20260927 22:43:41.957553 14899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:41.958094 14899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:41.958253 14899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3a8442c2c94bf9b2af1a3243038f3b, State: Initialized, Role: FOLLOWER
I20260927 22:43:41.958763 14751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 4
I20260927 22:43:41.959861 14899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:41.964175 14913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:41.965745 14913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:41.966751 14899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:41.975425 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.941640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.1.53.126:37717)
0927 22:43:41.942068 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.942076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.952812 (+ 10736us) server_negotiation.cc:407] Connection header received
0927 22:43:41.953129 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.953136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.953225 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.953304 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.956503 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.957743 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.960181 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.960944 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.961050 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.966192 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.966199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.966204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.966247 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.971620 (+ 5373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.972370 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.972379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.972385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.972572 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.973108 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.973116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.973120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.973487 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.973750 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.974767 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.975034 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
Sep 27 22:43:41 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:43:41.978773 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.941453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.1.53.124:35929)
0927 22:43:41.941840 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.941848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.953321 (+ 11473us) server_negotiation.cc:407] Connection header received
0927 22:43:41.953531 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.953536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.953626 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.953718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.956315 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.957377 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.961196 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.962944 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.963079 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.968069 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.968076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.968081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.968118 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.974710 (+ 6592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.975658 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.975663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.975667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.975849 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.976359 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.976364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.976367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.976773 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.977036 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.978165 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.978396 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260927 22:43:41.983731 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.941393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.1.53.125:46519)
0927 22:43:41.941630 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.941639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.952522 (+ 10883us) server_negotiation.cc:407] Connection header received
0927 22:43:41.953130 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.953136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.953225 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.953304 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.956508 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.957743 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.960493 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.961197 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.961301 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.969394 (+ 8093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.969403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.969410 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.969467 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.975483 (+ 6016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.978956 (+ 3473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.978967 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.978973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.979200 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.981335 (+ 2135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.981346 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.981353 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.981914 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.982275 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.983091 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.983338 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:43:41.985888 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.959372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54670)
0927 22:43:41.959710 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:41.959731 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:41.959833 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:41.964389 (+ 4556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:41.964395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:41.964947 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:41.965178 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.965191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.967683 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.967691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.969957 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:41.969974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.971050 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.971057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:41.971272 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.971898 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:41.971923 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:41.978759 (+ 6836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:41.983003 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.983008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.983011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.983478 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.984311 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:41.984317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:41.984321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:41.984470 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:41.985000 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:41.985005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:41.985190 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 22:43:41.985325 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:43:41.985628 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260927 22:43:41.986815 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:41.959398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.1.53.124:35929)
0927 22:43:41.963989 (+ 4591us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:41.963997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:41.964029 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:43:41.964094 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:41.964101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:41.964190 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:41.964349 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:41.966083 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.967489 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.970146 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:41.970859 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:41.971573 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:41.979209 (+ 7636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:41.979220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:41.979226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:41.979272 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:41.982831 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.983639 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.983647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.983650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.984163 (+ 513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:41.984582 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:41.984589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:41.984592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:41.984871 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:41.985029 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:41.986300 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0927 22:43:41.986467 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4416,"thread_start_us":4120,"threads_started":1}
I20260927 22:43:41.992092 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: Bootstrap starting.
I20260927 22:43:41.995342 14751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:41.996835 14751 log.cc:824] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: Log is configured to *not* fsync() on all Append() calls
W20260927 22:43:42.001754 14924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:43:42.001901 14924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:43:41.989624 14832 master_runner.cc:186] Error getting master registration for 127.1.53.126:37717: OK, instance_id { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" instance_seqno: 1790549020817609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:43:42.003265 14832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:43:42.006428 14751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: No bootstrap required, opened a new log
I20260927 22:43:42.008143 14913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:42.008503 14913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.009917 14913 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13de2c3352304577ab16e0d3852366b5 (127.1.53.126:37717), 87b73f0c472444518bd292ff69983648 (127.1.53.125:46519)
I20260927 22:43:42.017777 14751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.018186 14751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:42.018314 14751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13de2c3352304577ab16e0d3852366b5, State: Initialized, Role: FOLLOWER
I20260927 22:43:42.018981 14751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.020027 14927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.020522 14927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:42.021416 14751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:42.021490 14726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13de2c3352304577ab16e0d3852366b5" is_pre_election: true
I20260927 22:43:42.022109 14726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3a8442c2c94bf9b2af1a3243038f3b in term 0.
I20260927 22:43:42.022836 14849 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13de2c3352304577ab16e0d3852366b5, 8b3a8442c2c94bf9b2af1a3243038f3b; no voters:
I20260927 22:43:42.023376 14913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:43:42.023586 14913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:43:42.023718 14913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:42.027729 14913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.030337 14726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13de2c3352304577ab16e0d3852366b5"
I20260927 22:43:42.030820 14726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:42.031534 14913 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [CANDIDATE]: Term 1 election: Requested vote from peers 13de2c3352304577ab16e0d3852366b5 (127.1.53.126:37717), 87b73f0c472444518bd292ff69983648 (127.1.53.125:46519)
I20260927 22:43:42.035190 14726 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3a8442c2c94bf9b2af1a3243038f3b in term 1.
I20260927 22:43:42.035694 14849 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13de2c3352304577ab16e0d3852366b5, 8b3a8442c2c94bf9b2af1a3243038f3b; no voters:
I20260927 22:43:42.036280 14913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:42.036981 14913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [term 1 LEADER]: Becoming Leader. State: Replica: 8b3a8442c2c94bf9b2af1a3243038f3b, State: Running, Role: LEADER
I20260927 22:43:42.037762 14913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.045014 14939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3a8442c2c94bf9b2af1a3243038f3b. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.046613 14939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:42.040198 14684 master_runner.cc:186] Error getting master registration for 127.1.53.125:46519: OK, instance_id { permanent_uuid: "87b73f0c472444518bd292ff69983648" instance_seqno: 1790549021335984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:43:42.054364 14684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:43:42.055305 14941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 22:43:42.056715 14705 tablet.cc:2369] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:42.057858 14941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 22:43:42.066644 14938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:43:42.066793 14938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:43:42.067226 14726 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [term 1 FOLLOWER]: Refusing update from remote peer 8b3a8442c2c94bf9b2af1a3243038f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:42.067754 14825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } attempt: 5
I20260927 22:43:42.068289 14939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 22:43:42 dist-test-slave-pp80 krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:43:42.087525 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:42.068394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:35929 (local address 127.0.0.1:54676)
0927 22:43:42.068582 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:42.068608 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:42.068735 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:42.069254 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:42.069260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:42.069836 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:42.070075 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:42.070087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:42.071812 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:42.071818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:42.073361 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:42.073376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:42.074791 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:42.074798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:42.075035 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:42.075764 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:42.075794 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:42.079204 (+ 3410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:42.084029 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:42.084035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:42.084039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:42.084756 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:42.085276 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:42.085282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:42.085289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:42.085481 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:42.086352 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:42.086358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:42.086560 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 22:43:42.086724 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:43:42.087093 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 22:43:42.088539 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:42.068732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.1.53.124:35929)
0927 22:43:42.068914 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:42.068920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:42.068938 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:43:42.068996 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:42.069002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:42.069108 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:42.069213 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:42.070251 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:42.071501 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:42.073626 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:42.074336 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:42.074449 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:42.079634 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:42.079647 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:42.079652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:42.079685 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:42.083802 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:42.084944 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:42.084950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:42.084954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:42.085120 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:42.085631 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:42.085637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:42.085643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:42.086123 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:42.086283 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:42.087904 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0927 22:43:42.088171 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:43:42.089917 14853 tablet.cc:2369] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:42.095824 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: Bootstrap starting.
I20260927 22:43:42.100667 14825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:42.102355 14825 log.cc:824] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:42.106716 14927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3a8442c2c94bf9b2af1a3243038f3b. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.107023 14927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:42.111685 14944 mvcc.cc:205] Tried to move back new op lower bound from 7334088794376306688 to 7334088794273021952. Current Snapshot: MvccSnapshot[applied={T|T < 7334088794376306688}]
I20260927 22:43:42.124125 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: No bootstrap required, opened a new log
I20260927 22:43:42.125793 14913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.126180 14913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:42.129246 14927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.129509 14927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:42.133416 14941 catalog_manager.cc:1347] Generated new cluster ID: c35a5e300d324e08a2e5b7495d1d2fe7
I20260927 22:43:42.133538 14941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:43:42.160813 14825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.161314 14825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:42.161451 14825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b73f0c472444518bd292ff69983648, State: Initialized, Role: FOLLOWER
I20260927 22:43:42.164754 14825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } }
I20260927 22:43:42.166409 14949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.167393 14949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:42.168449 14825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:42.170951 14799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b73f0c472444518bd292ff69983648"
I20260927 22:43:42.171459 14799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:42.175119 14799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3a8442c2c94bf9b2af1a3243038f3b in term 1.
I20260927 22:43:42.179270 14798 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 1 FOLLOWER]: Refusing update from remote peer 8b3a8442c2c94bf9b2af1a3243038f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:42.180075 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b73f0c472444518bd292ff69983648" is_pre_election: true
I20260927 22:43:42.180459 14800 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8b3a8442c2c94bf9b2af1a3243038f3b in term 1. Re-sending same reply.
I20260927 22:43:42.182384 14939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:42.191432 14941 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:43:42.193615 14941 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:43:42.208308 14941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b: Generated new TSK 0
I20260927 22:43:42.209735 14941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:43:42.266520 14949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3a8442c2c94bf9b2af1a3243038f3b. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.266858 14949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:42.267509 14913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.267879 14913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:42.278815 14957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13de2c3352304577ab16e0d3852366b5" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37717 } } peers { permanent_uuid: "87b73f0c472444518bd292ff69983648" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 46519 } } peers { permanent_uuid: "8b3a8442c2c94bf9b2af1a3243038f3b" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 35929 } } }
I20260927 22:43:42.281674 14957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:43:42.325167 14966 catalog_manager.cc:1259] Loaded cluster ID: c35a5e300d324e08a2e5b7495d1d2fe7
I20260927 22:43:42.325367 14966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: loading cluster ID for follower catalog manager: success
I20260927 22:43:42.332803 14966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: acquiring CA information for follower catalog manager: success
I20260927 22:43:42.339910 14966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87b73f0c472444518bd292ff69983648: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:43:43.069172 14938 catalog_manager.cc:1259] Loaded cluster ID: c35a5e300d324e08a2e5b7495d1d2fe7
I20260927 22:43:43.069279 14938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: loading cluster ID for follower catalog manager: success
I20260927 22:43:43.071884 14938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: acquiring CA information for follower catalog manager: success
I20260927 22:43:43.073763 14938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13de2c3352304577ab16e0d3852366b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:43:43.387435 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14684
I20260927 22:43:43.388880 14746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:43.501008 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14684
I20260927 22:43:43.521844 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14758
I20260927 22:43:43.523042 14820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:43.634071 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14758
I20260927 22:43:43.634963 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:43.633491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37717 (local address 127.0.0.1:60036)
0927 22:43:43.634046 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:43.634201 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":331,"threads_started":1}
W20260927 22:43:43.635357 14849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.126:37717: connect: Connection refused (error 111)
I20260927 22:43:43.636885 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:43.636527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37717 (local address 127.0.0.1:60052)
0927 22:43:43.636686 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:43.636775 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:37717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:43:43.637151 14849 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b3a8442c2c94bf9b2af1a3243038f3b -> Peer 13de2c3352304577ab16e0d3852366b5 (127.1.53.126:37717): Couldn't send request to peer 13de2c3352304577ab16e0d3852366b5. Status: Network error: Client connection negotiation failed: client connection to 127.1.53.126:37717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:43:43.655336 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14832
I20260927 22:43:43.656570 14894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:43.768148 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14832
2026-09-27T22:43:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4768 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:43:43 dist-test-slave-pp80 krb5kdc[14977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:43:43 dist-test-slave-pp80 krb5kdc[14977](info): set up 2 sockets
Sep 27 22:43:43 dist-test-slave-pp80 krb5kdc[14977](info): commencing operation
krb5kdc: starting...
W20260927 22:43:44.872759 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:43:45 dist-test-slave-pp80 krb5kdc[14977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:43:45.117552 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:39427
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:41405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:39427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:45.454703 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:45.455013 14990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:45.455147 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:45.462599 14990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:45.462730 14990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:45.462777 14990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:45.462883 14990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:45.463068 14990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:45.473248 14990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:41405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:39427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:39427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:45.474599 14990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:45.475919 14990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:45.488003 14998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:45.487980 14996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:45.488160 14995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:45.488909 14990 server_base.cc:1034] running on GCE node
I20260927 22:43:45.489820 14990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:45.491401 14990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:45.492652 14990 hybrid_clock.cc:648] HybridClock initialized: now 1790549025492611 us; error 51 us; skew 500 ppm
I20260927 22:43:45.493044 14990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:45 dist-test-slave-pp80 krb5kdc[14977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:45.496223 14990 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:45.497761 14990 webserver.cc:460] Webserver started at http://127.1.53.126:44989/ using document root <none> and password file <none>
I20260927 22:43:45.498466 14990 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:45.498610 14990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:45.498929 14990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:45.501956 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "3b655d52fd834a8c9096124b939b20bb"
format_stamp: "Formatted at 2026-09-27 22:43:45 on dist-test-slave-pp80"
server_key: "7e52f8fb2f38759c09ebbc909fe6f5f6"
server_key_iv: "0cbd3213e3d913e08b4a47e1f671ca61"
server_key_version: "encryptionkey@0"
I20260927 22:43:45.502763 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "3b655d52fd834a8c9096124b939b20bb"
format_stamp: "Formatted at 2026-09-27 22:43:45 on dist-test-slave-pp80"
server_key: "7e52f8fb2f38759c09ebbc909fe6f5f6"
server_key_iv: "0cbd3213e3d913e08b4a47e1f671ca61"
server_key_version: "encryptionkey@0"
I20260927 22:43:45.508109 14990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:43:45.511647 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:45.513072 14990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:45.513255 14990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "3b655d52fd834a8c9096124b939b20bb"
format_stamp: "Formatted at 2026-09-27 22:43:45 on dist-test-slave-pp80"
server_key: "7e52f8fb2f38759c09ebbc909fe6f5f6"
server_key_iv: "0cbd3213e3d913e08b4a47e1f671ca61"
server_key_version: "encryptionkey@0"
I20260927 22:43:45.513464 14990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:45.529711 14990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:45.534637 14990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:45.534998 14990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:45.555891 14990 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:39427
I20260927 22:43:45.555915 15056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:39427 every 8 connection(s)
I20260927 22:43:45.557556 14990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:45.561846 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 14990
I20260927 22:43:45.562150 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:43:45.562354 15057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:45.562995 1237 external_mini_cluster.cc:1428] Setting key 5478d2d105125fb623c196bab5ccdfdc
I20260927 22:43:45.579157 15057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb: Bootstrap starting.
Sep 27 22:43:45 dist-test-slave-pp80 krb5kdc[14977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:45.584128 15057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:45.585414 15057 log.cc:824] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:45.588662 15057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb: No bootstrap required, opened a new log
I20260927 22:43:45.590528 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:45.565139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.1.53.126:39427)
0927 22:43:45.566081 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:45.566098 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:45.566159 (+ 61us) server_negotiation.cc:407] Connection header received
0927 22:43:45.568308 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:45.568349 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:45.568973 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:45.569527 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:45.571160 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:45.573422 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:45.574923 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:45.576208 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:45.576481 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:45.582043 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:45.582053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:45.582073 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:45.582129 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:45.585966 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:45.586805 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:45.586819 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:45.586826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:45.587005 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:45.587453 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:45.587460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:45.587464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:45.588051 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:45.588437 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:45.588945 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 22:43:45.589436 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":396,"threads_started":1}
I20260927 22:43:45.595134 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:43:45.598407 15057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b655d52fd834a8c9096124b939b20bb" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 39427 } }
I20260927 22:43:45.598879 15057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:45.599004 15057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b655d52fd834a8c9096124b939b20bb, State: Initialized, Role: FOLLOWER
I20260927 22:43:45.599659 15057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b655d52fd834a8c9096124b939b20bb" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 39427 } }
I20260927 22:43:45.599938 15057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:43:45.600103 15057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:43:45.600309 15057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:45.602672 15057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b655d52fd834a8c9096124b939b20bb" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 39427 } }
I20260927 22:43:45.603319 15057 leader_election.cc:302] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b655d52fd834a8c9096124b939b20bb; no voters:
I20260927 22:43:45.603979 15057 leader_election.cc:288] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:43:45.604149 15062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:45.605003 15062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [term 1 LEADER]: Becoming Leader. State: Replica: 3b655d52fd834a8c9096124b939b20bb, State: Running, Role: LEADER
I20260927 22:43:45.605575 15062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b655d52fd834a8c9096124b939b20bb" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 39427 } }
I20260927 22:43:45.606312 15057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:45.608660 15063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b655d52fd834a8c9096124b939b20bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b655d52fd834a8c9096124b939b20bb" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 39427 } } }
I20260927 22:43:45.608690 15064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b655d52fd834a8c9096124b939b20bb. Latest consensus state: current_term: 1 leader_uuid: "3b655d52fd834a8c9096124b939b20bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b655d52fd834a8c9096124b939b20bb" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 39427 } } }
I20260927 22:43:45.609145 15063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:45.609212 15064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:45.613082 15067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:43:45.618335 15067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:43:45.629354 15067 catalog_manager.cc:1347] Generated new cluster ID: c8f4d56f8c514da48d52c897d79745a7
I20260927 22:43:45.629510 15067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:43:45.637863 15067 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:43:45.639321 15067 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:43:45.659198 15067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b655d52fd834a8c9096124b939b20bb: Generated new TSK 0
I20260927 22:43:45.660284 15067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:43:45.707981 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 14990
I20260927 22:43:45.709276 15052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:45.806509 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 14990
2026-09-27T22:43:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2037 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:43:45 dist-test-slave-pp80 krb5kdc[15085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:43:45 dist-test-slave-pp80 krb5kdc[15085](info): set up 2 sockets
Sep 27 22:43:45 dist-test-slave-pp80 krb5kdc[15085](info): commencing operation
krb5kdc: starting...
W20260927 22:43:46.917092 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:43:47 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:43:47.165205 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42459
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:47.500713 15098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:47.501024 15098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:47.501116 15098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:47.508580 15098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:47.508692 15098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:47.508737 15098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:47.508791 15098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:47.508834 15098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:47.518759 15098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:47.520094 15098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:47.521518 15098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:47.533926 15104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:47.533907 15106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:47.534014 15103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:47.539923 15098 server_base.cc:1034] running on GCE node
I20260927 22:43:47.540943 15098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:47.542605 15098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:47.543895 15098 hybrid_clock.cc:648] HybridClock initialized: now 1790549027543832 us; error 47 us; skew 500 ppm
I20260927 22:43:47.544377 15098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:47 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549027, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:47.547776 15098 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:47.549654 15098 webserver.cc:460] Webserver started at http://127.1.53.126:33765/ using document root <none> and password file <none>
I20260927 22:43:47.550436 15098 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:47.550580 15098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:47.550895 15098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:47.554044 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "ef05dd017bef45dfa677614edd3bb17f"
format_stamp: "Formatted at 2026-09-27 22:43:47 on dist-test-slave-pp80"
server_key: "498baec8582e3e6e35150d1cedcd7ee5"
server_key_iv: "2fc5ef5c42efdf013170ee4f55ef335b"
server_key_version: "encryptionkey@0"
I20260927 22:43:47.554821 15098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "ef05dd017bef45dfa677614edd3bb17f"
format_stamp: "Formatted at 2026-09-27 22:43:47 on dist-test-slave-pp80"
server_key: "498baec8582e3e6e35150d1cedcd7ee5"
server_key_iv: "2fc5ef5c42efdf013170ee4f55ef335b"
server_key_version: "encryptionkey@0"
I20260927 22:43:47.560678 15098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 22:43:47.564805 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:47.566535 15098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:43:47.566756 15098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "ef05dd017bef45dfa677614edd3bb17f"
format_stamp: "Formatted at 2026-09-27 22:43:47 on dist-test-slave-pp80"
server_key: "498baec8582e3e6e35150d1cedcd7ee5"
server_key_iv: "2fc5ef5c42efdf013170ee4f55ef335b"
server_key_version: "encryptionkey@0"
I20260927 22:43:47.566996 15098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:47.597707 15098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:47.602352 15098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:47.602694 15098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:47.623570 15098 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42459
I20260927 22:43:47.623605 15164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42459 every 8 connection(s)
I20260927 22:43:47.625157 15098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:47.629068 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:47.629904 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15098
I20260927 22:43:47.630116 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:43:47.630820 1237 external_mini_cluster.cc:1428] Setting key 63a184e2720414441f3f2736c7e754cf
I20260927 22:43:47.643474 15165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap starting.
Sep 27 22:43:47 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:47.647503 15165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:47.648783 15165 log.cc:824] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:47.652819 15165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: No bootstrap required, opened a new log
I20260927 22:43:47.655977 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:47.633002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.1.53.126:42459)
0927 22:43:47.633916 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:47.633929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:47.633984 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:43:47.635705 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:47.635728 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:47.636154 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:47.636577 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:47.638342 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:47.639762 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:47.641727 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:47.642488 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:47.642660 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:47.647549 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:47.647558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:47.647595 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:47.647652 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:47.651212 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:47.652328 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:47.652337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:47.652350 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:47.652542 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:47.653062 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:47.653066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:47.653068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:47.653619 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:47.653868 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:47.654360 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 22:43:47.654783 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":448,"threads_started":1}
I20260927 22:43:47.662822 15165 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:47.663335 15165 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:47.663451 15165 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef05dd017bef45dfa677614edd3bb17f, State: Initialized, Role: FOLLOWER
I20260927 22:43:47.664012 15165 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:47.664268 15165 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:43:47.664425 15165 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:43:47.664601 15165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:47.666780 15165 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:47.667271 15165 leader_election.cc:302] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef05dd017bef45dfa677614edd3bb17f; no voters:
I20260927 22:43:47.667874 15165 leader_election.cc:288] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:43:47.668042 15170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:47.668776 15170 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 LEADER]: Becoming Leader. State: Replica: ef05dd017bef45dfa677614edd3bb17f, State: Running, Role: LEADER
I20260927 22:43:47.669462 15170 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:47.670221 15165 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:47.672670 15172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef05dd017bef45dfa677614edd3bb17f. Latest consensus state: current_term: 1 leader_uuid: "ef05dd017bef45dfa677614edd3bb17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } } }
I20260927 22:43:47.673023 15172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:47.672730 15171 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef05dd017bef45dfa677614edd3bb17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } } }
I20260927 22:43:47.673177 15171 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:47.674620 15175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:43:47.680676 15175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:43:47.690742 15175 catalog_manager.cc:1347] Generated new cluster ID: 2452633dabe244038451db683b6ba897
I20260927 22:43:47.690872 15175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:43:47.726298 15175 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:43:47.728057 15175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:43:47.734388 15175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Generated new TSK 0
I20260927 22:43:47.735478 15175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:43:47.792979 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42459
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:48.144285 15191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:48.144588 15191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:48.144686 15191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:48.152521 15191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:48.152647 15191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:48.152763 15191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:43:48.166589 15191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:48.168170 15191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:48.169672 15191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:48.183933 15199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:48.184067 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:48.184072 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:48.185038 15191 server_base.cc:1034] running on GCE node
I20260927 22:43:48.185717 15191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:48.186733 15191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:48.188004 15191 hybrid_clock.cc:648] HybridClock initialized: now 1790549028187928 us; error 63 us; skew 500 ppm
I20260927 22:43:48.188417 15191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:48 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:48.191820 15191 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:43:48.193636 15191 webserver.cc:460] Webserver started at http://127.1.53.65:39425/ using document root <none> and password file <none>
I20260927 22:43:48.194370 15191 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:48.194521 15191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:48.194849 15191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:48.197911 15191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "2f4e1228bce04d2ebd66955e9877b58a"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "a2ebee4b6c2614f79bccbaed5df817f8"
server_key_iv: "c439eef54dbea4acfd2eae83bf376328"
server_key_version: "encryptionkey@0"
I20260927 22:43:48.198658 15191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "2f4e1228bce04d2ebd66955e9877b58a"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "a2ebee4b6c2614f79bccbaed5df817f8"
server_key_iv: "c439eef54dbea4acfd2eae83bf376328"
server_key_version: "encryptionkey@0"
I20260927 22:43:48.203886 15191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:43:48.207597 15206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:48.209347 15191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:48.209551 15191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "2f4e1228bce04d2ebd66955e9877b58a"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "a2ebee4b6c2614f79bccbaed5df817f8"
server_key_iv: "c439eef54dbea4acfd2eae83bf376328"
server_key_version: "encryptionkey@0"
I20260927 22:43:48.209769 15191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:48.236900 15191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:48.241564 15191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:48.241858 15191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:48.242836 15191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:48.244313 15191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:48.244407 15191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:48.244556 15191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:48.244623 15191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:48.284878 15191 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:38745
I20260927 22:43:48.284965 15318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:38745 every 8 connection(s)
I20260927 22:43:48.286257 15191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:43:48.286790 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15191
I20260927 22:43:48.287070 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:43:48.287973 1237 external_mini_cluster.cc:1428] Setting key 88c1c461460c3eddb1e690c777d23dd2
Sep 27 22:43:48 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:48.307770 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:48.289154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:49329 (local address 127.1.53.126:42459)
0927 22:43:48.289646 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:48.289652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:48.291085 (+ 1433us) server_negotiation.cc:407] Connection header received
0927 22:43:48.292746 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:48.292751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:48.292842 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:48.292951 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:48.294456 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.295177 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.296841 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.297288 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.297427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:48.301449 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:48.301454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:48.301457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:48.301492 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:48.304124 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:48.305183 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:48.305187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:48.305189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:48.305314 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:48.305863 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:48.305867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:48.305869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:48.306200 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:48.306364 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:48.307145 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 22:43:48.307375 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260927 22:43:48.308900 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:48.289796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:49329)
0927 22:43:48.290634 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:48.290686 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:48.292473 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:48.293199 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:48.293207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:48.293736 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:48.294287 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:48.294308 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.295338 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.295348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:48.296692 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:48.296704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.297545 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.297553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:48.297788 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:48.298481 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:48.298512 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:48.301167 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:48.304308 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:48.304321 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:48.304343 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:48.305044 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:48.305431 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:48.305439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:48.305445 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:48.305728 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:48.306352 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:48.306361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:48.306745 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 22:43:48.307033 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 22:43:48.307650 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":346,"threads_started":1}
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:43:48.310842 15319 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42459
I20260927 22:43:48.311127 15319 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:48.311946 15319 heartbeater.cc:511] Master 127.1.53.126:42459 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
I20260927 22:43:48.314455 15130 ts_manager.cc:194] Registered new tserver with Master: 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65:38745)
I20260927 22:43:48.315922 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:49329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:43:48.343278 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42459
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:48.668401 15325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:48.668694 15325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:48.668838 15325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:48.676209 15325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:48.676389 15325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:48.676580 15325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:43:48.686658 15325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:48.687932 15325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:48.689302 15325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:48.702214 15331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:48.702135 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:48.702176 15330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:48.702735 15325 server_base.cc:1034] running on GCE node
I20260927 22:43:48.703358 15325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:48.704474 15325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:48.705708 15325 hybrid_clock.cc:648] HybridClock initialized: now 1790549028705656 us; error 44 us; skew 500 ppm
I20260927 22:43:48.706045 15325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:48 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:48.709267 15325 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:43:48.710824 15325 webserver.cc:460] Webserver started at http://127.1.53.66:33159/ using document root <none> and password file <none>
I20260927 22:43:48.711532 15325 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:48.711669 15325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:48.711964 15325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:48.715068 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "2eefe8f5b8de448e85872deddfdf7858"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "80cb2fd6a62129b15095fab81d5c1950"
server_key_iv: "b7830771b4d15ca20e0190d5fbdd0cf4"
server_key_version: "encryptionkey@0"
I20260927 22:43:48.715811 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "2eefe8f5b8de448e85872deddfdf7858"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "80cb2fd6a62129b15095fab81d5c1950"
server_key_iv: "b7830771b4d15ca20e0190d5fbdd0cf4"
server_key_version: "encryptionkey@0"
I20260927 22:43:48.720968 15325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:43:48.724452 15340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:48.725878 15325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:43:48.726063 15325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "2eefe8f5b8de448e85872deddfdf7858"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "80cb2fd6a62129b15095fab81d5c1950"
server_key_iv: "b7830771b4d15ca20e0190d5fbdd0cf4"
server_key_version: "encryptionkey@0"
I20260927 22:43:48.726253 15325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:48.747303 15325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:48.751739 15325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:48.752031 15325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:48.753051 15325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:48.754400 15325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:48.754505 15325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:48.754660 15325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:48.754729 15325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:48.788841 15325 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:42735
I20260927 22:43:48.788898 15452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:42735 every 8 connection(s)
I20260927 22:43:48.790203 15325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:43:48.796181 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15325
I20260927 22:43:48.796392 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:43:48.796832 1237 external_mini_cluster.cc:1428] Setting key aae105fc8c0b039b7abfd0923776337a
Sep 27 22:43:48 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:48.811465 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:48.792877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:40361 (local address 127.1.53.126:42459)
0927 22:43:48.793090 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:48.793095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:48.794314 (+ 1219us) server_negotiation.cc:407] Connection header received
0927 22:43:48.796201 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:48.796207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:48.796343 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:48.796471 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:48.798242 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.799067 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.800708 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.801138 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.801294 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:48.805046 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:48.805052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:48.805055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:48.805087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:48.807941 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:48.808880 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:48.808884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:48.808886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:48.809009 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:48.809595 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:48.809603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:48.809608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:48.809932 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:48.810079 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:48.811010 (+ 931us) server_negotiation.cc:299] Negotiation successful
0927 22:43:48.811211 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":92}
I20260927 22:43:48.813163 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:48.793374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:40361)
0927 22:43:48.794127 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:48.794166 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:48.795918 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:48.796724 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:48.796733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:48.797358 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:48.798074 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:48.798093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.799221 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.799225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:48.800564 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:48.800577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:48.801381 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:48.801388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:48.801628 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:48.802358 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:48.802381 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:48.804782 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:48.808076 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:48.808087 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:48.808107 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:48.808769 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:48.809159 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:48.809167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:48.809173 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:48.809466 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:48.810063 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:48.810075 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:48.810613 (+ 538us) client_negotiation.cc:769] Sending connection context
0927 22:43:48.810917 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 22:43:48.811528 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":289,"threads_started":1}
I20260927 22:43:48.814814 15453 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42459
I20260927 22:43:48.815150 15453 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:48.816027 15453 heartbeater.cc:511] Master 127.1.53.126:42459 requested a full tablet report, sending...
I20260927 22:43:48.817750 15130 ts_manager.cc:194] Registered new tserver with Master: 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
I20260927 22:43:48.818578 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:40361
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:43:48.855620 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42459
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:49.175913 15458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:49.176195 15458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:49.176410 15458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:49.184130 15458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:49.184288 15458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:49.184433 15458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:43:49.194447 15458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:49.195750 15458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:49.197125 15458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:49.209964 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:49.209985 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:49.209923 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:49.210803 15458 server_base.cc:1034] running on GCE node
I20260927 22:43:49.211512 15458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:49.212476 15458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:49.213735 15458 hybrid_clock.cc:648] HybridClock initialized: now 1790549029213681 us; error 44 us; skew 500 ppm
I20260927 22:43:49.214113 15458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:49 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:49.217456 15458 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:43:49.218955 15458 webserver.cc:460] Webserver started at http://127.1.53.67:36989/ using document root <none> and password file <none>
I20260927 22:43:49.219655 15458 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:49.219796 15458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:49.220103 15458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:43:49.223176 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "19a9be5563ed426bbe6daabf56fc18ca"
format_stamp: "Formatted at 2026-09-27 22:43:49 on dist-test-slave-pp80"
server_key: "713fdd8fb027b7db3c263da878226413"
server_key_iv: "2ca1bfab6d0c905daf03a0e5bd05745e"
server_key_version: "encryptionkey@0"
I20260927 22:43:49.223965 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "19a9be5563ed426bbe6daabf56fc18ca"
format_stamp: "Formatted at 2026-09-27 22:43:49 on dist-test-slave-pp80"
server_key: "713fdd8fb027b7db3c263da878226413"
server_key_iv: "2ca1bfab6d0c905daf03a0e5bd05745e"
server_key_version: "encryptionkey@0"
I20260927 22:43:49.229723 15458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:43:49.233665 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:49.235122 15458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:43:49.235311 15458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "19a9be5563ed426bbe6daabf56fc18ca"
format_stamp: "Formatted at 2026-09-27 22:43:49 on dist-test-slave-pp80"
server_key: "713fdd8fb027b7db3c263da878226413"
server_key_iv: "2ca1bfab6d0c905daf03a0e5bd05745e"
server_key_version: "encryptionkey@0"
I20260927 22:43:49.235524 15458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:49.247821 15458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:49.251713 15458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:49.252008 15458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:49.252971 15458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:49.254436 15458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:49.254535 15458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:49.254683 15458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:49.254751 15458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:49.289826 15458 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39009
I20260927 22:43:49.289860 15585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39009 every 8 connection(s)
I20260927 22:43:49.291102 15458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:43:49.297979 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15458
I20260927 22:43:49.298162 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:43:49.298614 1237 external_mini_cluster.cc:1428] Setting key 5b15f7a59a0d9df1160c178252084e39
Sep 27 22:43:49 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:49.311167 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.293722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:38385 (local address 127.1.53.126:42459)
0927 22:43:49.293945 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.293951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.295155 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 22:43:49.297118 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.297123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.297220 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.297311 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:49.298933 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.299876 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.301678 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.302095 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.302226 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.305655 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:49.305660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:49.305664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:49.305696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:49.308203 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:49.308944 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:49.308950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:49.308954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:49.309081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:49.309494 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:49.309499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:49.309503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:49.309768 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:49.309923 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.310597 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.310812 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:43:49.312193 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.294239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:38385)
0927 22:43:49.294993 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:49.295030 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:49.296788 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:49.297530 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:49.297538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:49.298065 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:49.298751 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.298777 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.300079 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.300084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.301534 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.301546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.302311 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.302319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.302564 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.303236 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:49.303258 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:49.305455 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:49.308385 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:49.308393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:49.308409 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:49.308816 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:49.309187 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:49.309193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:49.309196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:49.309375 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:49.309894 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:49.309902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:49.310223 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 22:43:49.310475 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:43:49.311009 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":315,"threads_started":1}
I20260927 22:43:49.313773 15586 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42459
I20260927 22:43:49.314152 15586 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:49.314954 15586 heartbeater.cc:511] Master 127.1.53.126:42459 requested a full tablet report, sending...
I20260927 22:43:49.316474 15129 ts_manager.cc:194] Registered new tserver with Master: 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009)
I20260927 22:43:49.317344 15129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:38385
I20260927 22:43:49.318723 15319 heartbeater.cc:503] Master 127.1.53.126:42459 was elected leader, sending a full tablet report...
I20260927 22:43:49.326538 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:43:49.343686 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.331114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.1.53.126:42459)
0927 22:43:49.331305 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.331312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.331747 (+ 435us) server_negotiation.cc:407] Connection header received
0927 22:43:49.331866 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.331870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.331926 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.332057 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:49.333375 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.334096 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.335463 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.335910 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.336020 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.338426 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:49.338431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:49.338434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:49.338464 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:49.341334 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:49.341998 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:49.342003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:49.342006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:49.342134 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:49.342424 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:49.342427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:49.342429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:49.342747 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:49.342945 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.343226 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.343384 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:43:49.347895 15129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:43:49.354992 15129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:43:49.383749 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.368496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.1.53.67:39009)
0927 22:43:49.375147 (+ 6651us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.375157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.375193 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:43:49.375302 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.375310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.375560 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.376147 (+ 587us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:49.376999 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.378088 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.380864 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.382414 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.382577 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.382844 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.383199 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.383373 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6467,"thread_start_us":6279,"threads_started":1}
I20260927 22:43:49.383854 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.368101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39009 (local address 127.0.0.1:34126)
0927 22:43:49.370026 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:49.370091 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:49.370294 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:49.375852 (+ 5558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:49.375860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:49.375884 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:49.376786 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.376801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.378216 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.378223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.380679 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.380698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.382627 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.382634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.382852 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.382935 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 22:43:49.383101 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:43:49.383321 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1582,"mutex_wait_us":25,"thread_start_us":1434,"threads_started":1}
I20260927 22:43:49.384773 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.371243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.1.53.65:38745)
0927 22:43:49.374075 (+ 2832us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.374089 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.375002 (+ 913us) server_negotiation.cc:407] Connection header received
0927 22:43:49.375133 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.375163 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.375400 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.375595 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:49.376962 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.378736 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.381329 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.382524 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.382651 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.382865 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.384158 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.384364 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2575,"thread_start_us":2350,"threads_started":1}
I20260927 22:43:49.385215 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.370655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:38646)
0927 22:43:49.374731 (+ 4076us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:49.374754 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:49.374873 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:49.375861 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:49.375902 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:49.375935 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:49.376757 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.376775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.378867 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.378874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.381186 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.381196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.383019 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.383025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.383199 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.383230 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:43:49.383302 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 22:43:49.383429 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3925,"mutex_wait_us":44,"spinlock_wait_cycles":768,"thread_start_us":2157,"threads_started":1}
I20260927 22:43:49.385622 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.368589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:42735 (local address 127.0.0.1:48126)
0927 22:43:49.373231 (+ 4642us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:49.373257 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:49.373989 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:49.375599 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:49.375610 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:49.375644 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:49.376562 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.376578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.378813 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.378818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.381883 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.381901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.383912 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.383920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.384415 (+ 495us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.384457 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:43:49.384534 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:43:49.384652 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4423,"mutex_wait_us":78,"thread_start_us":5786,"threads_started":1}
I20260927 22:43:49.385721 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.369214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.1.53.66:42735)
0927 22:43:49.372569 (+ 3355us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.372583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.373374 (+ 791us) server_negotiation.cc:407] Connection header received
0927 22:43:49.374292 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.374306 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.374634 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.375294 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:49.376762 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.378641 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.382072 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.383448 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.383577 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.383756 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.385061 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.385237 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3111,"thread_start_us":1508,"threads_started":1}
I20260927 22:43:49.388001 15521 tablet_service.cc:1467] Processing CreateTablet for tablet fbf1a51e97c94ee1b8d394040bb72287 (DEFAULT_TABLE table=test-table [id=9ea07a9ec0bf4f6595c2a4c915647e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:49.389308 15521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf1a51e97c94ee1b8d394040bb72287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:49.389261 15254 tablet_service.cc:1467] Processing CreateTablet for tablet fbf1a51e97c94ee1b8d394040bb72287 (DEFAULT_TABLE table=test-table [id=9ea07a9ec0bf4f6595c2a4c915647e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:49.389261 15388 tablet_service.cc:1467] Processing CreateTablet for tablet fbf1a51e97c94ee1b8d394040bb72287 (DEFAULT_TABLE table=test-table [id=9ea07a9ec0bf4f6595c2a4c915647e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:49.390738 15388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf1a51e97c94ee1b8d394040bb72287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:49.390763 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf1a51e97c94ee1b8d394040bb72287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:49.400874 15604 tablet_bootstrap.cc:492] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Bootstrap starting.
I20260927 22:43:49.404285 15604 tablet_bootstrap.cc:654] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:49.405460 15604 log.cc:824] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:49.408521 15606 tablet_bootstrap.cc:492] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Bootstrap starting.
I20260927 22:43:49.408982 15604 tablet_bootstrap.cc:492] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: No bootstrap required, opened a new log
I20260927 22:43:49.409341 15604 ts_tablet_manager.cc:1397] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260927 22:43:49.412175 15606 tablet_bootstrap.cc:654] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:49.412477 15607 tablet_bootstrap.cc:492] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Bootstrap starting.
I20260927 22:43:49.413465 15606 log.cc:824] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:49.416189 15607 tablet_bootstrap.cc:654] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:49.416826 15606 tablet_bootstrap.cc:492] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: No bootstrap required, opened a new log
I20260927 22:43:49.417124 15606 ts_tablet_manager.cc:1397] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 22:43:49.417447 15607 log.cc:824] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:49.419983 15604 raft_consensus.cc:348] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.420452 15604 raft_consensus.cc:374] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:49.420550 15604 raft_consensus.cc:729] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a9be5563ed426bbe6daabf56fc18ca, State: Initialized, Role: FOLLOWER
I20260927 22:43:49.421247 15604 consensus_queue.cc:260] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.422526 15586 heartbeater.cc:503] Master 127.1.53.126:42459 was elected leader, sending a full tablet report...
I20260927 22:43:49.424492 15604 ts_tablet_manager.cc:1428] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260927 22:43:49.426837 15607 tablet_bootstrap.cc:492] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: No bootstrap required, opened a new log
I20260927 22:43:49.427132 15607 ts_tablet_manager.cc:1397] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260927 22:43:49.428107 15606 raft_consensus.cc:348] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.428606 15606 raft_consensus.cc:374] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:49.428711 15606 raft_consensus.cc:729] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4e1228bce04d2ebd66955e9877b58a, State: Initialized, Role: FOLLOWER
I20260927 22:43:49.429384 15606 consensus_queue.cc:260] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.431151 15606 ts_tablet_manager.cc:1428] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260927 22:43:49.435855 15607 raft_consensus.cc:348] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.436208 15607 raft_consensus.cc:374] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:49.436347 15607 raft_consensus.cc:729] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eefe8f5b8de448e85872deddfdf7858, State: Initialized, Role: FOLLOWER
I20260927 22:43:49.436856 15607 consensus_queue.cc:260] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.437785 15453 heartbeater.cc:503] Master 127.1.53.126:42459 was elected leader, sending a full tablet report...
I20260927 22:43:49.438431 15607 ts_tablet_manager.cc:1428] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260927 22:43:49.538841 15320 tablet.cc:2369] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:43:49.542194 15454 tablet.cc:2369] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:43:49.542840 15587 tablet.cc:2369] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:49.789443 15611 raft_consensus.cc:482] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:49.789793 15611 raft_consensus.cc:504] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.791590 15611 leader_election.cc:288] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
I20260927 22:43:49.800230 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.792196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:33687 (local address 127.1.53.67:39009)
0927 22:43:49.792451 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.792458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.793156 (+ 698us) server_negotiation.cc:407] Connection header received
0927 22:43:49.793450 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.793456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.793522 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.793635 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:49.794492 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.795557 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.798302 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.799183 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.799351 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.799473 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.799798 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.799940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:43:49.800413 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.791955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39009 (local address 127.1.53.65:33687)
0927 22:43:49.793037 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:49.793060 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:49.793272 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:49.793641 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:49.793645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:49.793902 (+ 257us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:49.794312 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.794329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.795696 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.795703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.798076 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.798090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.799413 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.799419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.799619 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.799646 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:43:49.799758 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:43:49.799909 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":34,"thread_start_us":689,"threads_started":1}
I20260927 22:43:49.801446 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.791955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:42735 (local address 127.1.53.65:44763)
0927 22:43:49.792816 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:49.792837 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:49.793038 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:49.793591 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:49.793596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:49.793621 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:49.794082 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.794099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.795309 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.795318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.798074 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:49.798090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.800185 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.800192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:49.800803 (+ 611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.800841 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:43:49.800940 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:43:49.801075 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":33,"thread_start_us":390,"threads_started":1}
I20260927 22:43:49.801478 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.792707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:44763 (local address 127.1.53.66:42735)
0927 22:43:49.792924 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.792930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.792968 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:43:49.793222 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.793228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.793304 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.793366 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:49.794262 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.795167 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.798279 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.799765 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.799908 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.800000 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.801044 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.801202 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:43:49.801734 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf1a51e97c94ee1b8d394040bb72287" candidate_uuid: "2f4e1228bce04d2ebd66955e9877b58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9be5563ed426bbe6daabf56fc18ca" is_pre_election: true
I20260927 22:43:49.802330 15541 raft_consensus.cc:2454] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4e1228bce04d2ebd66955e9877b58a in term 0.
I20260927 22:43:49.802866 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf1a51e97c94ee1b8d394040bb72287" candidate_uuid: "2f4e1228bce04d2ebd66955e9877b58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eefe8f5b8de448e85872deddfdf7858" is_pre_election: true
I20260927 22:43:49.803015 15207 leader_election.cc:302] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a9be5563ed426bbe6daabf56fc18ca, 2f4e1228bce04d2ebd66955e9877b58a; no voters:
I20260927 22:43:49.803429 15611 raft_consensus.cc:2781] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:43:49.803385 15408 raft_consensus.cc:2454] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4e1228bce04d2ebd66955e9877b58a in term 0.
I20260927 22:43:49.803593 15611 raft_consensus.cc:482] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:43:49.803707 15611 raft_consensus.cc:3037] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:49.806416 15611 raft_consensus.cc:504] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.807152 15611 leader_election.cc:288] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 1 election: Requested vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
I20260927 22:43:49.807564 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf1a51e97c94ee1b8d394040bb72287" candidate_uuid: "2f4e1228bce04d2ebd66955e9877b58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9be5563ed426bbe6daabf56fc18ca"
I20260927 22:43:49.807653 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf1a51e97c94ee1b8d394040bb72287" candidate_uuid: "2f4e1228bce04d2ebd66955e9877b58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eefe8f5b8de448e85872deddfdf7858"
I20260927 22:43:49.807785 15541 raft_consensus.cc:3037] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:49.807878 15408 raft_consensus.cc:3037] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:49.810118 15408 raft_consensus.cc:2454] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4e1228bce04d2ebd66955e9877b58a in term 1.
I20260927 22:43:49.810271 15541 raft_consensus.cc:2454] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4e1228bce04d2ebd66955e9877b58a in term 1.
I20260927 22:43:49.810637 15209 leader_election.cc:302] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eefe8f5b8de448e85872deddfdf7858, 2f4e1228bce04d2ebd66955e9877b58a; no voters:
I20260927 22:43:49.810958 15611 raft_consensus.cc:2781] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:49.811425 15611 raft_consensus.cc:686] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 1 LEADER]: Becoming Leader. State: Replica: 2f4e1228bce04d2ebd66955e9877b58a, State: Running, Role: LEADER
I20260927 22:43:49.811882 15611 consensus_queue.cc:237] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:49.816582 15129 catalog_manager.cc:5495] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65). New cstate: current_term: 1 leader_uuid: "2f4e1228bce04d2ebd66955e9877b58a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } health_report { overall_health: HEALTHY } } }
I20260927 22:43:49.834712 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.829155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.1.53.65:38745)
0927 22:43:49.829365 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.829373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.829397 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:43:49.829638 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.829642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.829745 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.829869 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:43:49.830258 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.831059 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.832557 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.832968 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.833066 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.833547 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:43:49.834109 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:43:49.834224 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.834322 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.834391 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:43:49.843725 15541 raft_consensus.cc:1261] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Refusing update from remote peer 2f4e1228bce04d2ebd66955e9877b58a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:49.843725 15408 raft_consensus.cc:1261] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Refusing update from remote peer 2f4e1228bce04d2ebd66955e9877b58a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:49.844838 15615 consensus_queue.cc:1035] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:49.845212 15611 consensus_queue.cc:1035] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:49.859302 15620 mvcc.cc:205] Tried to move back new op lower bound from 7334088826228125696 to 7334088826114211840. Current Snapshot: MvccSnapshot[applied={T|T < 7334088826228125696}]
I20260927 22:43:49.873003 15129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:43:49.873294 15129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:43:49.877243 15129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Sending DeleteTablet for 3 replicas of tablet fbf1a51e97c94ee1b8d394040bb72287
I20260927 22:43:49.878813 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15191
I20260927 22:43:49.878772 15521 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf1a51e97c94ee1b8d394040bb72287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:49 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:34126
I20260927 22:43:49.878793 15254 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf1a51e97c94ee1b8d394040bb72287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:49 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:38646
I20260927 22:43:49.879592 15388 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf1a51e97c94ee1b8d394040bb72287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:49 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:48126
I20260927 22:43:49.892598 15629 tablet_replica.cc:330] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: stopping tablet replica
I20260927 22:43:49.893117 15629 raft_consensus.cc:2229] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:43:49.893684 15629 raft_consensus.cc:2258] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:49.894579 15631 tablet_replica.cc:330] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: stopping tablet replica
I20260927 22:43:49.895044 15631 raft_consensus.cc:2229] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:43:49.895566 15631 raft_consensus.cc:2258] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:43:49 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:43:49.897997 15628 tablet_replica.cc:330] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: stopping tablet replica
I20260927 22:43:49.898028 15631 ts_tablet_manager.cc:1905] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:49.898517 15628 raft_consensus.cc:2229] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:43:49.899289 15628 raft_consensus.cc:2258] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:49.901173 15629 ts_tablet_manager.cc:1905] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:49.902283 15628 ts_tablet_manager.cc:1905] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:49.908435 15629 ts_tablet_manager.cc:1918] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:49.908612 15629 log.cc:1196] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/fbf1a51e97c94ee1b8d394040bb72287
I20260927 22:43:49.909197 15629 ts_tablet_manager.cc:1939] T fbf1a51e97c94ee1b8d394040bb72287 P 19a9be5563ed426bbe6daabf56fc18ca: Deleting consensus metadata
I20260927 22:43:49.910878 15628 ts_tablet_manager.cc:1918] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:49.911158 15628 log.cc:1196] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/fbf1a51e97c94ee1b8d394040bb72287
I20260927 22:43:49.911213 15114 catalog_manager.cc:4841] TS 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): tablet fbf1a51e97c94ee1b8d394040bb72287 (table test-table [id=9ea07a9ec0bf4f6595c2a4c915647e32]) successfully deleted
I20260927 22:43:49.911839 15628 ts_tablet_manager.cc:1939] T fbf1a51e97c94ee1b8d394040bb72287 P 2f4e1228bce04d2ebd66955e9877b58a: Deleting consensus metadata
I20260927 22:43:49.912214 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:49.880041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.1.53.65:38745)
0927 22:43:49.880258 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:49.880265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:49.880513 (+ 248us) server_negotiation.cc:407] Connection header received
0927 22:43:49.880873 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:49.880887 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:49.881023 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:49.881115 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:49.883741 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.885470 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.888299 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:49.889359 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:49.889495 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:49.897773 (+ 8278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:49.897790 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:49.897811 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:49.897887 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:49.903367 (+ 5480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:49.908743 (+ 5376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:49.908755 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:49.908759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:49.908993 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:49.909686 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:49.909692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:49.909696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:49.910189 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:49.910497 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:49.911400 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 22:43:49.911732 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:43:49.913597 15314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:49.915307 15631 ts_tablet_manager.cc:1918] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:49.915542 15631 log.cc:1196] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/fbf1a51e97c94ee1b8d394040bb72287
I20260927 22:43:49.916208 15631 ts_tablet_manager.cc:1939] T fbf1a51e97c94ee1b8d394040bb72287 P 2eefe8f5b8de448e85872deddfdf7858: Deleting consensus metadata
I20260927 22:43:49.918325 15116 catalog_manager.cc:4841] TS 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): tablet fbf1a51e97c94ee1b8d394040bb72287 (table test-table [id=9ea07a9ec0bf4f6595c2a4c915647e32]) successfully deleted
I20260927 22:43:50.015499 15114 catalog_manager.cc:4841] TS 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65:38745): tablet fbf1a51e97c94ee1b8d394040bb72287 (table test-table [id=9ea07a9ec0bf4f6595c2a4c915647e32]) successfully deleted
I20260927 22:43:50.016819 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15191
I20260927 22:43:50.035487 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15325
Sep 27 22:43:50 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:43:50.051388 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:50.036286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.1.53.66:42735)
0927 22:43:50.036464 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:50.036471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:50.036567 (+ 96us) server_negotiation.cc:407] Connection header received
0927 22:43:50.036773 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:50.036779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:50.036853 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:50.037010 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:50.038253 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:50.039010 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:50.040322 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:50.040734 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:50.040877 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:50.045224 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:50.045239 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:50.045254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:50.045287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:50.048585 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:50.049295 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:50.049302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:50.049305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:50.049437 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:50.049812 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:50.049817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:50.049819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:50.050212 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:50.050508 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:50.050816 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 22:43:50.051048 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:43:50.052151 15448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:50.146384 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15325
I20260927 22:43:50.163712 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15458
Sep 27 22:43:50 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:43:50.182040 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:50.164592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.1.53.67:39009)
0927 22:43:50.164811 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:50.164816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:50.164836 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:43:50.165036 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:50.165041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:50.165125 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:50.165225 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:50.166606 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:50.167334 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:50.169138 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:50.169828 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:50.170014 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:50.175619 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:50.175638 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:50.175654 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:50.175702 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:50.178792 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:50.179660 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:50.179671 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:50.179673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:50.179921 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:50.180514 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:50.180518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:50.180520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:50.180911 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:50.181157 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:50.181467 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 22:43:50.181642 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 22:43:50.182746 15581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:50.279218 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15458
I20260927 22:43:50.297001 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15098
I20260927 22:43:50.298300 15160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:50.411125 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15098
I20260927 22:43:50.432997 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42459
--webserver_interface=127.1.53.126
--webserver_port=33765
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:50.765975 15636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:50.766275 15636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:50.766444 15636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:50.773569 15636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:50.773676 15636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:50.773758 15636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:50.773844 15636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:50.773952 15636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:50.783622 15636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=33765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:50.784967 15636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:50.786280 15636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:50.809144 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:50.809092 15645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:50.809172 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:50.810561 15636 server_base.cc:1034] running on GCE node
I20260927 22:43:50.811431 15636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:50.813113 15636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:50.814330 15636 hybrid_clock.cc:648] HybridClock initialized: now 1790549030814267 us; error 59 us; skew 500 ppm
I20260927 22:43:50.814782 15636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:50 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549030, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:50.818612 15636 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:50.820221 15636 webserver.cc:460] Webserver started at http://127.1.53.126:33765/ using document root <none> and password file <none>
I20260927 22:43:50.821041 15636 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:50.821156 15636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:50.827912 15636 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 22:43:50.831324 15652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:50.832962 15636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:43:50.833164 15636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "ef05dd017bef45dfa677614edd3bb17f"
format_stamp: "Formatted at 2026-09-27 22:43:47 on dist-test-slave-pp80"
server_key: "498baec8582e3e6e35150d1cedcd7ee5"
server_key_iv: "2fc5ef5c42efdf013170ee4f55ef335b"
server_key_version: "encryptionkey@0"
I20260927 22:43:50.834297 15636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:50.851231 15636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:50.856062 15636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:50.856416 15636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:50.876641 15636 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42459
I20260927 22:43:50.876740 15703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42459 every 8 connection(s)
I20260927 22:43:50.878268 15636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:50.886163 15704 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:43:50.886615 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15636
I20260927 22:43:50.887400 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:38745
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:42459
--webserver_port=39425
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:50.889819 15704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap starting.
I20260927 22:43:50.915160 15704 log.cc:824] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:50.931209 15704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:43:50.931857 15704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap complete.
I20260927 22:43:50.945326 15704 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:50.946530 15704 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef05dd017bef45dfa677614edd3bb17f, State: Initialized, Role: FOLLOWER
I20260927 22:43:50.947305 15704 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:50.947633 15704 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:43:50.947779 15704 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:43:50.947997 15704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:43:50.954025 15704 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:50.954667 15704 leader_election.cc:302] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef05dd017bef45dfa677614edd3bb17f; no voters:
I20260927 22:43:50.955325 15704 leader_election.cc:288] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:43:50.955505 15708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:43:50.957008 15708 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 LEADER]: Becoming Leader. State: Replica: ef05dd017bef45dfa677614edd3bb17f, State: Running, Role: LEADER
I20260927 22:43:50.957616 15708 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:50.957760 15704 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:50.960201 15710 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef05dd017bef45dfa677614edd3bb17f. Latest consensus state: current_term: 2 leader_uuid: "ef05dd017bef45dfa677614edd3bb17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } } }
I20260927 22:43:50.960628 15710 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:50.960264 15709 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef05dd017bef45dfa677614edd3bb17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } } }
I20260927 22:43:50.961182 15709 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:50.962989 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:43:50.971638 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:43:50.973119 15715 catalog_manager.cc:1259] Loaded cluster ID: 2452633dabe244038451db683b6ba897
I20260927 22:43:50.973254 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:43:50.976476 15715 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:43:50.994879 15715 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:43:51.004848 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:43:51.006647 15715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Loaded TSK: 0
I20260927 22:43:51.008147 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:43:51.313633 15706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:51.313922 15706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:51.314069 15706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:51.321765 15706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:51.321877 15706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:51.322041 15706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:43:51.332351 15706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:38745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=39425
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:51.333673 15706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:51.334997 15706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:51.351167 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:51.351388 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:51.351562 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:51.354177 15706 server_base.cc:1034] running on GCE node
I20260927 22:43:51.354913 15706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:51.356127 15706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:51.357366 15706 hybrid_clock.cc:648] HybridClock initialized: now 1790549031357316 us; error 45 us; skew 500 ppm
I20260927 22:43:51.357751 15706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:51 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:51.361583 15706 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:43:51.363461 15706 webserver.cc:460] Webserver started at http://127.1.53.65:39425/ using document root <none> and password file <none>
I20260927 22:43:51.364430 15706 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:51.364599 15706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:51.370749 15706 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 22:43:51.373876 15741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:51.375454 15706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:43:51.375623 15706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "2f4e1228bce04d2ebd66955e9877b58a"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "a2ebee4b6c2614f79bccbaed5df817f8"
server_key_iv: "c439eef54dbea4acfd2eae83bf376328"
server_key_version: "encryptionkey@0"
I20260927 22:43:51.376741 15706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:51.389503 15706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:51.393898 15706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:51.394171 15706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:51.395242 15706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:51.396714 15706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:51.396801 15706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:51.396946 15706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:51.397040 15706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:51.431738 15706 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:38745
I20260927 22:43:51.431831 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:38745 every 8 connection(s)
I20260927 22:43:51.433245 15706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:43:51.440016 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15706
I20260927 22:43:51.440685 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:42735
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:42459
--webserver_port=33159
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:43:51 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:51.461514 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:51.436486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:60159)
0927 22:43:51.437275 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:51.437318 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:51.439126 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:51.440576 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:51.440586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:51.441202 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:51.441844 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:51.441868 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.444342 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.444352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:51.446546 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:51.446565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.448414 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.448424 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:51.448729 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:51.449524 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:51.449557 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:51.452296 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:51.457019 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:51.457028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:51.457044 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:51.457470 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:51.457980 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:51.457983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:51.457985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:51.458213 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:51.459270 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:51.459279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:51.459698 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 22:43:51.460004 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 22:43:51.460455 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":307,"threads_started":1}
I20260927 22:43:51.462173 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:51.436124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:60159 (local address 127.1.53.126:42459)
0927 22:43:51.436754 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:51.436774 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:51.437445 (+ 671us) server_negotiation.cc:407] Connection header received
0927 22:43:51.439600 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:51.439612 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:51.439916 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:51.440419 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:51.442102 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.444100 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.446737 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.447802 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.447987 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:51.452594 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:51.452608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:51.452630 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:51.452685 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:51.456842 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:51.457633 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:51.457644 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:51.457654 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:51.457864 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:51.458341 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:51.458349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:51.458355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:51.459041 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:51.459252 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:51.460317 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0927 22:43:51.460823 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":266,"threads_started":1}
I20260927 22:43:51.465179 15854 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42459
I20260927 22:43:51.465534 15854 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:51.466493 15854 heartbeater.cc:511] Master 127.1.53.126:42459 requested a full tablet report, sending...
I20260927 22:43:51.468966 15669 ts_manager.cc:194] Registered new tserver with Master: 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65:38745)
I20260927 22:43:51.470438 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:60159
W20260927 22:43:51.776578 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:51.776856 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:51.776953 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:51.784711 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:51.784839 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:51.784973 15858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:43:51.795679 15858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:42735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=33159
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:51.797022 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:51.798273 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:51.811406 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:51.811558 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:51.811579 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:51.812441 15858 server_base.cc:1034] running on GCE node
I20260927 22:43:51.813030 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:51.813970 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:51.815193 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790549031815139 us; error 49 us; skew 500 ppm
I20260927 22:43:51.815507 15858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:51 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:51.818862 15858 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:43:51.820429 15858 webserver.cc:460] Webserver started at http://127.1.53.66:33159/ using document root <none> and password file <none>
I20260927 22:43:51.821107 15858 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:51.821255 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:51.826572 15858 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 22:43:51.829815 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:51.831368 15858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:43:51.831573 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "2eefe8f5b8de448e85872deddfdf7858"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "80cb2fd6a62129b15095fab81d5c1950"
server_key_iv: "b7830771b4d15ca20e0190d5fbdd0cf4"
server_key_version: "encryptionkey@0"
I20260927 22:43:51.832768 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:51.849136 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:51.853808 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:51.854130 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:51.855289 15858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:51.856803 15858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:51.856945 15858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:51.857076 15858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:51.857168 15858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:51.894819 15858 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:42735
I20260927 22:43:51.894872 15985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:42735 every 8 connection(s)
I20260927 22:43:51.896478 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:43:51.898387 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15858
I20260927 22:43:51.899219 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:39009
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:42459
--webserver_port=36989
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:43:51 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549031, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:51.920313 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:51.901498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:43027 (local address 127.1.53.126:42459)
0927 22:43:51.901753 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:51.901760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:51.901787 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:43:51.903141 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:51.903148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:51.903276 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:51.903397 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:51.905094 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.906260 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.908738 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.909267 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.909390 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:51.913859 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:51.913865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:51.913868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:51.913906 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:51.916871 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:51.917749 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:51.917755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:51.917759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:51.917908 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:51.918382 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:51.918387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:51.918391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:51.918703 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:51.918869 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:51.919612 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 22:43:51.919893 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:43:51.921373 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:51.899957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:43027)
0927 22:43:51.900812 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:51.900860 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:51.902827 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:51.903663 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:51.903673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:51.904298 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:51.904895 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:51.904924 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.906441 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.906445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:51.908529 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:51.908549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:51.909520 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:51.909530 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:51.909858 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:51.910658 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:51.910692 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:51.913594 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:51.917034 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:51.917044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:51.917062 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:51.917611 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:51.918037 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:51.918043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:51.918047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:51.918253 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:51.918840 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:51.918848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:51.919218 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 22:43:51.919507 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 22:43:51.920126 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":338,"threads_started":1}
I20260927 22:43:51.923043 15986 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42459
I20260927 22:43:51.923363 15986 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:51.924144 15986 heartbeater.cc:511] Master 127.1.53.126:42459 requested a full tablet report, sending...
I20260927 22:43:51.925737 15669 ts_manager.cc:194] Registered new tserver with Master: 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
I20260927 22:43:51.926894 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:43027
W20260927 22:43:52.238001 15989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:52.238329 15989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:52.238499 15989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:52.245918 15989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:52.246032 15989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:52.246209 15989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:43:52.259841 15989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:39009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=36989
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:52.261718 15989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:52.263589 15989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:52.278936 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:52.278957 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:52.278936 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:52.279526 15989 server_base.cc:1034] running on GCE node
I20260927 22:43:52.280277 15989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:52.281296 15989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:52.282522 15989 hybrid_clock.cc:648] HybridClock initialized: now 1790549032282422 us; error 85 us; skew 500 ppm
I20260927 22:43:52.282917 15989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:52 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549032, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:52.286540 15989 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:43:52.287957 15989 webserver.cc:460] Webserver started at http://127.1.53.67:36989/ using document root <none> and password file <none>
I20260927 22:43:52.288757 15989 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:52.288909 15989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:52.294153 15989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:43:52.296967 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:52.298345 15989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:52.298573 15989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "19a9be5563ed426bbe6daabf56fc18ca"
format_stamp: "Formatted at 2026-09-27 22:43:49 on dist-test-slave-pp80"
server_key: "713fdd8fb027b7db3c263da878226413"
server_key_iv: "2ca1bfab6d0c905daf03a0e5bd05745e"
server_key_version: "encryptionkey@0"
I20260927 22:43:52.299664 15989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:52.314261 15989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:52.319209 15989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:52.319507 15989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:52.320562 15989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:52.321951 15989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:52.322088 15989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:52.322224 15989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:52.322322 15989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:52.359624 15989 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39009
I20260927 22:43:52.359689 16116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39009 every 8 connection(s)
I20260927 22:43:52.361394 15989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:43:52.365474 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 15989
Sep 27 22:43:52 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549032, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:43:52.384629 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.366575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.1.53.126:42459)
0927 22:43:52.367138 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.367148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.368966 (+ 1818us) server_negotiation.cc:407] Connection header received
0927 22:43:52.369007 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.369012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.369102 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.369216 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:52.370891 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.372194 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.373812 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.374311 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.374459 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.377848 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:52.377854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:52.377858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:52.377892 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:52.381541 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.382371 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.382378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.382387 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.382531 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.382926 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.382933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.382936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.383247 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:52.383544 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.383862 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.384110 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260927 22:43:52.385232 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.364348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:40305 (local address 127.1.53.126:42459)
0927 22:43:52.364596 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.364604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.366331 (+ 1727us) server_negotiation.cc:407] Connection header received
0927 22:43:52.368607 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.368613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.368712 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.368830 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:52.370755 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.372194 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.373892 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.374397 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.374546 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.378396 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:52.378405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:52.378408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:52.378448 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:52.381759 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.382704 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.382712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.382716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.382842 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.383300 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.383308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.383313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.383608 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:52.383763 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.384601 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.384836 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":68}
I20260927 22:43:52.386516 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.364967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:40305)
0927 22:43:52.366075 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.366170 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:52.368309 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:52.369165 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:52.369180 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:52.369861 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:52.370566 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.370586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.372387 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.372392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.373724 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.373741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.374643 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.374651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.374886 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.375583 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:52.375605 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:52.378161 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:52.382044 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:52.382053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:52.382068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:52.382553 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:52.382968 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:52.382973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:52.382977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:52.383180 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:52.383743 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:52.383752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:52.384123 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 22:43:52.384491 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 22:43:52.385090 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":484,"threads_started":1}
I20260927 22:43:52.387902 16117 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42459
I20260927 22:43:52.388281 16117 heartbeater.cc:464] Registering TS with master...
I20260927 22:43:52.389050 16117 heartbeater.cc:511] Master 127.1.53.126:42459 requested a full tablet report, sending...
I20260927 22:43:52.390538 15669 ts_manager.cc:194] Registered new tserver with Master: 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009)
I20260927 22:43:52.391358 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:40305
I20260927 22:43:52.400352 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:43:52.407902 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.401284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.1.53.126:42459)
0927 22:43:52.401509 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.401520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.401931 (+ 411us) server_negotiation.cc:407] Connection header received
0927 22:43:52.402150 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.402154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.402293 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.402424 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:43:52.402947 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.403822 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.405483 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.405978 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.406090 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.406694 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:43:52.407211 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:43:52.407325 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.407497 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.407571 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:43:52.408970 15669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:43:52.416399 15669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:43:52.449425 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.438883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.1.53.65:38745)
0927 22:43:52.440191 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.440200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.440232 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:43:52.440355 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.440362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.440627 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.441529 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:52.441793 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.443635 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.446418 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.447547 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.447702 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.447963 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.448660 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.448861 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":917,"threads_started":1}
I20260927 22:43:52.449476 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.429075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39009 (local address 127.0.0.1:44892)
0927 22:43:52.436528 (+ 7453us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.436573 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:52.436796 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:52.438972 (+ 2176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:52.438983 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:52.439029 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:52.439563 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.439576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.442586 (+ 3010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.442592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.445979 (+ 3387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.445996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.448231 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.448265 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.448441 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.448479 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:43:52.448854 (+ 375us) client_negotiation.cc:240] Negotiation successful
0927 22:43:52.449024 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7169,"mutex_wait_us":1,"thread_start_us":9001,"threads_started":1}
I20260927 22:43:52.450012 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.429279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.1.53.67:39009)
0927 22:43:52.437840 (+ 8561us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.437851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.437887 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:43:52.438097 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.438108 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.438421 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.438655 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:52.439793 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.442394 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.446168 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.447655 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.447788 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.448054 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.449433 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.449654 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8363,"thread_start_us":8193,"threads_started":1}
I20260927 22:43:52.450500 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.438734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53838)
0927 22:43:52.439398 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.439419 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:52.439559 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:52.440866 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:52.440872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:52.440901 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:52.441298 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.441314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.443823 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.443829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.446267 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.446277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.447772 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.447780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.448005 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.448076 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 22:43:52.448246 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:43:52.450018 (+ 1772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":95,"thread_start_us":2499,"threads_started":1}
I20260927 22:43:52.453240 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.429259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:42735 (local address 127.0.0.1:44922)
0927 22:43:52.438351 (+ 9092us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.438378 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:52.438505 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:52.445362 (+ 6857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:52.445366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:52.445393 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:52.445847 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.445870 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.447920 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.447925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.450396 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.450409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.452311 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.452319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.452499 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.452534 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:43:52.452654 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 22:43:52.452791 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8961,"mutex_wait_us":47,"thread_start_us":6840,"threads_started":1}
I20260927 22:43:52.453270 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.429857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.1.53.66:42735)
0927 22:43:52.444716 (+ 14859us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.444725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.444756 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:43:52.444874 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.444881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.445117 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.446029 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:52.446291 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.447777 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.450572 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.452046 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.452208 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.452463 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.452698 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.452873 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14694,"thread_start_us":15403,"threads_started":1}
I20260927 22:43:52.454274 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a (DEFAULT_TABLE table=test-table [id=7f33ab8421724550ae6491e6c9e62778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:52.454274 15789 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a (DEFAULT_TABLE table=test-table [id=7f33ab8421724550ae6491e6c9e62778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:52.455813 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:52.455829 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:52.456712 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a (DEFAULT_TABLE table=test-table [id=7f33ab8421724550ae6491e6c9e62778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:43:52.457983 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:43:52.471659 16133 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Bootstrap starting.
I20260927 22:43:52.473829 16134 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Bootstrap starting.
I20260927 22:43:52.473883 15854 heartbeater.cc:503] Master 127.1.53.126:42459 was elected leader, sending a full tablet report...
I20260927 22:43:52.476357 16135 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Bootstrap starting.
I20260927 22:43:52.478802 16133 tablet_bootstrap.cc:654] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:52.479070 16134 tablet_bootstrap.cc:654] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:52.480134 16134 log.cc:824] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:52.480208 16135 tablet_bootstrap.cc:654] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Neither blocks nor log segments found. Creating new log.
I20260927 22:43:52.480376 16133 log.cc:824] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:52.481861 16135 log.cc:824] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:52.483628 16134 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: No bootstrap required, opened a new log
I20260927 22:43:52.483922 16134 ts_tablet_manager.cc:1397] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:43:52.484685 16133 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: No bootstrap required, opened a new log
I20260927 22:43:52.485021 16133 ts_tablet_manager.cc:1397] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260927 22:43:52.486866 16135 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: No bootstrap required, opened a new log
I20260927 22:43:52.487147 16135 ts_tablet_manager.cc:1397] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 22:43:52.491315 16134 raft_consensus.cc:348] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.491683 16134 raft_consensus.cc:374] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:52.491770 16134 raft_consensus.cc:729] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4e1228bce04d2ebd66955e9877b58a, State: Initialized, Role: FOLLOWER
I20260927 22:43:52.492409 16134 consensus_queue.cc:260] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.494165 16134 ts_tablet_manager.cc:1428] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260927 22:43:52.496840 16133 raft_consensus.cc:348] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.497367 16133 raft_consensus.cc:374] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:52.497478 16133 raft_consensus.cc:729] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eefe8f5b8de448e85872deddfdf7858, State: Initialized, Role: FOLLOWER
I20260927 22:43:52.498160 16133 consensus_queue.cc:260] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.499272 15986 heartbeater.cc:503] Master 127.1.53.126:42459 was elected leader, sending a full tablet report...
I20260927 22:43:52.499028 16135 raft_consensus.cc:348] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.499460 16135 raft_consensus.cc:374] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:43:52.499576 16135 raft_consensus.cc:729] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a9be5563ed426bbe6daabf56fc18ca, State: Initialized, Role: FOLLOWER
I20260927 22:43:52.500159 16133 ts_tablet_manager.cc:1428] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260927 22:43:52.500293 16135 consensus_queue.cc:260] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.501578 16117 heartbeater.cc:503] Master 127.1.53.126:42459 was elected leader, sending a full tablet report...
I20260927 22:43:52.502234 16135 ts_tablet_manager.cc:1428] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
W20260927 22:43:52.613233 16118 tablet.cc:2369] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:43:52.648653 15987 tablet.cc:2369] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:52.682030 16140 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:52.682343 16140 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.684587 16140 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65:38745)
W20260927 22:43:52.685590 15855 tablet.cc:2369] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:43:52.692052 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.685032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:57101 (local address 127.1.53.67:39009)
0927 22:43:52.685245 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.685250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.685918 (+ 668us) server_negotiation.cc:407] Connection header received
0927 22:43:52.686236 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.686242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.686323 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.686405 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:52.687255 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.688147 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.690153 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.691042 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.691138 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.691213 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.691654 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.691801 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:43:52.692262 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.686136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:60179 (local address 127.1.53.65:38745)
0927 22:43:52.686327 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.686333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.686738 (+ 405us) server_negotiation.cc:407] Connection header received
0927 22:43:52.686855 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.686861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.686941 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.687043 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:43:52.687676 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.688592 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.690436 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.691299 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.691433 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.691530 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.691866 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.691999 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:43:52.692159 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.685036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39009 (local address 127.1.53.66:57101)
0927 22:43:52.685806 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.685829 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:52.686043 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:52.686626 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:52.686629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:52.686649 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:52.687101 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.687114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.688315 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.688321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.690003 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.690015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.691268 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.691278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.691478 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.691515 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:43:52.691611 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:43:52.691707 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":22,"spinlock_wait_cycles":24320,"thread_start_us":441,"threads_started":1}
I20260927 22:43:52.692626 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.685961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.1.53.66:60179)
0927 22:43:52.686623 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.686638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:52.686749 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:52.687079 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:52.687082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:52.687101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:43:52.687503 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.687514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.688729 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.688733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.690280 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:52.690294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.691505 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.691509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:52.691682 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.691714 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:43:52.691815 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:43:52.691951 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1,"thread_start_us":367,"threads_started":1}
I20260927 22:43:52.693357 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ad4a7b9ef4444ba45178d7cb3346a" candidate_uuid: "2eefe8f5b8de448e85872deddfdf7858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9be5563ed426bbe6daabf56fc18ca" is_pre_election: true
I20260927 22:43:52.693578 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ad4a7b9ef4444ba45178d7cb3346a" candidate_uuid: "2eefe8f5b8de448e85872deddfdf7858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4e1228bce04d2ebd66955e9877b58a" is_pre_election: true
I20260927 22:43:52.693971 16072 raft_consensus.cc:2454] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eefe8f5b8de448e85872deddfdf7858 in term 0.
I20260927 22:43:52.694084 15809 raft_consensus.cc:2454] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eefe8f5b8de448e85872deddfdf7858 in term 0.
I20260927 22:43:52.694689 15874 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eefe8f5b8de448e85872deddfdf7858, 2f4e1228bce04d2ebd66955e9877b58a; no voters:
I20260927 22:43:52.695034 16140 raft_consensus.cc:2781] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:43:52.695181 16140 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:43:52.695286 16140 raft_consensus.cc:3037] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:52.697670 16140 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.698263 16140 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [CANDIDATE]: Term 1 election: Requested vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65:38745)
I20260927 22:43:52.698801 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ad4a7b9ef4444ba45178d7cb3346a" candidate_uuid: "2eefe8f5b8de448e85872deddfdf7858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9be5563ed426bbe6daabf56fc18ca"
I20260927 22:43:52.698791 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0ad4a7b9ef4444ba45178d7cb3346a" candidate_uuid: "2eefe8f5b8de448e85872deddfdf7858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4e1228bce04d2ebd66955e9877b58a"
I20260927 22:43:52.699002 16072 raft_consensus.cc:3037] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:52.699020 15809 raft_consensus.cc:3037] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:43:52.701270 16072 raft_consensus.cc:2454] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eefe8f5b8de448e85872deddfdf7858 in term 1.
I20260927 22:43:52.701598 15809 raft_consensus.cc:2454] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eefe8f5b8de448e85872deddfdf7858 in term 1.
I20260927 22:43:52.701839 15874 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a9be5563ed426bbe6daabf56fc18ca, 2eefe8f5b8de448e85872deddfdf7858; no voters:
I20260927 22:43:52.702241 16140 raft_consensus.cc:2781] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:43:52.702732 16140 raft_consensus.cc:686] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 1 LEADER]: Becoming Leader. State: Replica: 2eefe8f5b8de448e85872deddfdf7858, State: Running, Role: LEADER
I20260927 22:43:52.703203 16140 consensus_queue.cc:237] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:52.708521 15668 catalog_manager.cc:5495] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66). New cstate: current_term: 1 leader_uuid: "2eefe8f5b8de448e85872deddfdf7858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } health_report { overall_health: UNKNOWN } } }
I20260927 22:43:52.754722 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.749176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.1.53.66:42735)
0927 22:43:52.749380 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.749385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.749403 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:43:52.749497 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.749500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.749583 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.749714 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:43:52.750127 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.750911 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.752488 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.753004 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.753144 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.753639 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:43:52.754208 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:43:52.754338 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.754409 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.754485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:43:52.763738 16072 raft_consensus.cc:1261] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Refusing update from remote peer 2eefe8f5b8de448e85872deddfdf7858: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:52.763981 15809 raft_consensus.cc:1261] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Refusing update from remote peer 2eefe8f5b8de448e85872deddfdf7858: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:43:52.765064 16140 consensus_queue.cc:1035] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:52.765420 16144 consensus_queue.cc:1035] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:43:52.777467 16147 mvcc.cc:205] Tried to move back new op lower bound from 7334088838189481984 to 7334088837956530176. Current Snapshot: MvccSnapshot[applied={T|T < 7334088838189481984}]
I20260927 22:43:52.792567 15669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:43:52.792914 15669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:43:52.796782 15669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Sending DeleteTablet for 3 replicas of tablet 6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:43:52.798102 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:52 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44892
I20260927 22:43:52.798097 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:52 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:44922
I20260927 22:43:52.798220 15789 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:43:52 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:53838
I20260927 22:43:52.799671 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15706
I20260927 22:43:52.808844 16158 tablet_replica.cc:330] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: stopping tablet replica
I20260927 22:43:52.809468 16158 raft_consensus.cc:2229] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:43:52.810256 16158 raft_consensus.cc:2258] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:52.814002 16159 tablet_replica.cc:330] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: stopping tablet replica
I20260927 22:43:52.814519 16159 raft_consensus.cc:2229] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:43:52.815157 16159 raft_consensus.cc:2258] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:52.817354 16158 ts_tablet_manager.cc:1905] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:52.825794 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.801792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.1.53.65:38745)
0927 22:43:52.802048 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.802057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.802084 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:43:52.802160 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.802170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.802306 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.802444 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:52.805265 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.807001 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.810117 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.811363 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.811543 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.816469 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:52.816485 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:52.816504 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:52.816549 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:52.820933 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.822023 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.822035 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.822039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.822232 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.822840 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.822849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.822854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.823372 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:52.823693 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.825094 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.825369 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:43:52.827056 15849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:52.828483 16157 tablet_replica.cc:330] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: stopping tablet replica
I20260927 22:43:52.829170 16157 raft_consensus.cc:2229] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:43:52.829895 16157 raft_consensus.cc:2258] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:43:52.826553 16158 ts_tablet_manager.cc:1918] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:52.830649 16158 log.cc:1196] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:43:52.832347 16158 ts_tablet_manager.cc:1939] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2eefe8f5b8de448e85872deddfdf7858: Deleting consensus metadata
I20260927 22:43:52.832782 16157 ts_tablet_manager.cc:1905] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:43:52.835098 15655 catalog_manager.cc:4841] TS 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): tablet 6b0ad4a7b9ef4444ba45178d7cb3346a (table test-table [id=7f33ab8421724550ae6491e6c9e62778]) successfully deleted
I20260927 22:43:52.840230 16157 ts_tablet_manager.cc:1918] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:43:52.840677 16157 log.cc:1196] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:43:52.841270 16157 ts_tablet_manager.cc:1939] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 19a9be5563ed426bbe6daabf56fc18ca: Deleting consensus metadata
I20260927 22:43:52.843202 15653 catalog_manager.cc:4841] TS 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): tablet 6b0ad4a7b9ef4444ba45178d7cb3346a (table test-table [id=7f33ab8421724550ae6491e6c9e62778]) successfully deleted
I20260927 22:43:52.963233 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15706
I20260927 22:43:52.981483 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15858
W20260927 22:43:52.981624 15653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:38745 (error 108)
I20260927 22:43:52.983965 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.983161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53862)
0927 22:43:52.983628 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:52.983754 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
W20260927 22:43:52.984274 15653 catalog_manager.cc:4584] TS 2f4e1228bce04d2ebd66955e9877b58a (127.1.53.65:38745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0ad4a7b9ef4444ba45178d7cb3346a: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
I20260927 22:43:52.998189 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:52.982682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.1.53.66:42735)
0927 22:43:52.982860 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:52.982866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:52.982886 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:43:52.982957 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:52.982963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:52.983041 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:52.983165 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:52.984964 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.986234 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.987763 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:52.988559 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:52.988696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:52.991287 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:52.991309 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:52.991328 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:52.991372 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:52.995390 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.996051 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.996061 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.996065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.996270 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:52.996660 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:52.996666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:52.996670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:52.997102 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:52.997400 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:52.997630 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:43:52.997827 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:43:52.998992 15981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:53.037876 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.037389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53866)
0927 22:43:53.037625 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.037730 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:43:53.039029 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.038619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53882)
0927 22:43:53.038791 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.038857 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:43:53.099526 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.099051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53892)
0927 22:43:53.099285 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.099370 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:43:53.100788 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.100396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53906)
0927 22:43:53.100545 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.100625 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 22:43:53.115563 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15858
I20260927 22:43:53.134958 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15989
I20260927 22:43:53.149101 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.135718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.1.53.67:39009)
0927 22:43:53.135913 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:53.135919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:53.136028 (+ 109us) server_negotiation.cc:407] Connection header received
0927 22:43:53.136216 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:53.136220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:53.136482 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:53.136589 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:53.137994 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:53.138757 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:53.140084 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:53.140557 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:53.140688 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:53.143229 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:53.143243 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:53.143266 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:53.143318 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:53.146259 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:53.146978 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:53.146991 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:53.146993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:53.147145 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:53.147527 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:53.147532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:53.147536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:53.147905 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:53.148214 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:53.148529 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:43:53.148743 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:43:53.149812 16112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:53.182116 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.181433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53916)
0927 22:43:53.181749 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.181888 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:43:53.183423 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.182943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53928)
0927 22:43:53.183202 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.183281 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 22:43:53.241767 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15989
I20260927 22:43:53.259344 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 15636
I20260927 22:43:53.260598 15699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:43:53.376482 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:53.375683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38745 (local address 127.0.0.1:53940)
0927 22:43:53.376108 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:53.376272 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 22:43:53.376487 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 15636
I20260927 22:43:53.398641 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42459
--webserver_interface=127.1.53.126
--webserver_port=33765
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:53.727062 16166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:53.727360 16166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:53.727504 16166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:53.734900 16166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:43:53.735013 16166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:53.735110 16166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:53.735206 16166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:43:53.735265 16166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:43:53.745070 16166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=33765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:53.746520 16166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:53.747826 16166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:53.760031 16171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:53.760103 16174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:53.760075 16172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:53.760941 16166 server_base.cc:1034] running on GCE node
I20260927 22:43:53.761783 16166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:53.763373 16166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:53.764585 16166 hybrid_clock.cc:648] HybridClock initialized: now 1790549033764528 us; error 50 us; skew 500 ppm
I20260927 22:43:53.764931 16166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:53 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549033, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:53.768558 16166 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:43:53.770188 16166 webserver.cc:460] Webserver started at http://127.1.53.126:33765/ using document root <none> and password file <none>
I20260927 22:43:53.770959 16166 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:53.771104 16166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:53.777083 16166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:43:53.780179 16181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:53.781708 16166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:43:53.781909 16166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "ef05dd017bef45dfa677614edd3bb17f"
format_stamp: "Formatted at 2026-09-27 22:43:47 on dist-test-slave-pp80"
server_key: "498baec8582e3e6e35150d1cedcd7ee5"
server_key_iv: "2fc5ef5c42efdf013170ee4f55ef335b"
server_key_version: "encryptionkey@0"
I20260927 22:43:53.783012 16166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:53.806598 16166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:53.811878 16166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:53.812206 16166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:53.832993 16166 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42459
I20260927 22:43:53.833011 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42459 every 8 connection(s)
I20260927 22:43:53.834606 16166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:43:53.842041 16233 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:43:53.842092 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16166
I20260927 22:43:53.842950 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:38745
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:42459
--webserver_port=39425
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:53.845614 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap starting.
I20260927 22:43:53.874856 16233 log.cc:824] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:53.898402 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:43:53.898964 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f: Bootstrap complete.
I20260927 22:43:53.908126 16233 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:53.909108 16233 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef05dd017bef45dfa677614edd3bb17f, State: Initialized, Role: FOLLOWER
I20260927 22:43:53.909758 16233 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:53.909999 16233 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:43:53.910123 16233 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:43:53.910297 16233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:43:53.914261 16233 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:53.914754 16233 leader_election.cc:302] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef05dd017bef45dfa677614edd3bb17f; no voters:
I20260927 22:43:53.915403 16233 leader_election.cc:288] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:43:53.915614 16237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:43:53.916724 16237 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [term 3 LEADER]: Becoming Leader. State: Replica: ef05dd017bef45dfa677614edd3bb17f, State: Running, Role: LEADER
I20260927 22:43:53.917436 16233 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:43:53.917410 16237 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } }
I20260927 22:43:53.921577 16238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef05dd017bef45dfa677614edd3bb17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } } }
I20260927 22:43:53.921962 16238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:53.922535 16239 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef05dd017bef45dfa677614edd3bb17f. Latest consensus state: current_term: 3 leader_uuid: "ef05dd017bef45dfa677614edd3bb17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef05dd017bef45dfa677614edd3bb17f" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42459 } } }
I20260927 22:43:53.922832 16239 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef05dd017bef45dfa677614edd3bb17f [sys.catalog]: This master's current role is: LEADER
I20260927 22:43:53.943993 16248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:43:53.957258 16248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:43:53.958935 16248 catalog_manager.cc:1259] Loaded cluster ID: 2452633dabe244038451db683b6ba897
I20260927 22:43:53.959101 16248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:43:53.961304 16248 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f40b41eb9ad google::LogMessage::Fail() at ??:0
@ 0x7f40b41ee88a google::LogMessage::SendToLog() at ??:0
@ 0x7f40b41eb4de google::LogMessage::Flush() at ??:0
@ 0x7f40b41eca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f40c9c1e512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f40c9c1b976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f40c9ca3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f40c9ca3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f40b6e37345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f40b6e3a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f40b6e3a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f40b6e15eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f40ba0ef6db start_thread at ??:0
@ 0x7f40b308561f clone at ??:0
W20260927 22:43:54.292748 16235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:54.293052 16235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:54.293236 16235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:54.301469 16235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:54.301594 16235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:54.301847 16235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:43:54.312391 16235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:38745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=39425
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:54.313802 16235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:54.315284 16235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:54.328786 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:54.328939 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:54.329242 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:54.330281 16235 server_base.cc:1034] running on GCE node
I20260927 22:43:54.330997 16235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:54.332191 16235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:54.333498 16235 hybrid_clock.cc:648] HybridClock initialized: now 1790549034333435 us; error 48 us; skew 500 ppm
I20260927 22:43:54.333848 16235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:54 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:54.337076 16235 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:43:54.338521 16235 webserver.cc:460] Webserver started at http://127.1.53.65:39425/ using document root <none> and password file <none>
I20260927 22:43:54.339169 16235 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:54.339264 16235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:54.344669 16235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:54.347633 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:54.349082 16235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:54.349207 16235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "2f4e1228bce04d2ebd66955e9877b58a"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "a2ebee4b6c2614f79bccbaed5df817f8"
server_key_iv: "c439eef54dbea4acfd2eae83bf376328"
server_key_version: "encryptionkey@0"
I20260927 22:43:54.350119 16235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:54.375639 16235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:54.380060 16235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:54.380405 16235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:54.381510 16235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:54.383402 16292 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 22:43:54.389474 16235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 22:43:54.389602 16235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260927 22:43:54.389700 16235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 22:43:54.392935 16235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 22:43:54.393019 16235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:54.393425 16292 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Bootstrap starting.
I20260927 22:43:54.401856 16292 log.cc:824] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:43:54.417764 16292 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:43:54.418797 16292 tablet_bootstrap.cc:492] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Bootstrap complete.
I20260927 22:43:54.419385 16292 ts_tablet_manager.cc:1397] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.004s
I20260927 22:43:54.430034 16292 raft_consensus.cc:348] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:54.431892 16292 raft_consensus.cc:729] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4e1228bce04d2ebd66955e9877b58a, State: Initialized, Role: FOLLOWER
I20260927 22:43:54.433164 16292 consensus_queue.cc:260] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:54.434937 16292 ts_tablet_manager.cc:1428] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 22:43:54.438309 16235 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:38745
I20260927 22:43:54.438344 16400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:38745 every 8 connection(s)
I20260927 22:43:54.439486 16235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:43:54.442946 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16235
I20260927 22:43:54.443869 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:42735
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:42459
--webserver_port=33159
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:43:54.445142 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:54.442861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:44579)
0927 22:43:54.443877 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:54.444185 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":471,"threads_started":1}
W20260927 22:43:54.445809 16401 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:42459 (0 consecutive failures): Network error: Failed to ping master at 127.1.53.126:42459: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
I20260927 22:43:54.446713 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:54.446402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:42653)
0927 22:43:54.446547 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:54.446613 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:43:54.447600 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:54.447285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:47509)
0927 22:43:54.447427 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:54.447494 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:43:54.447847 16401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:43:54.770171 16404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:54.770465 16404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:54.770617 16404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:54.777993 16404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:54.778105 16404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:54.778292 16404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:43:54.788220 16404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:42735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=33159
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:54.789583 16404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:54.791110 16404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:54.804623 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:54.804661 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:54.804702 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:54.805481 16404 server_base.cc:1034] running on GCE node
I20260927 22:43:54.806141 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:54.807085 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:54.808331 16404 hybrid_clock.cc:648] HybridClock initialized: now 1790549034808278 us; error 42 us; skew 500 ppm
I20260927 22:43:54.808679 16404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:54 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:54.811859 16404 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:43:54.813318 16404 webserver.cc:460] Webserver started at http://127.1.53.66:33159/ using document root <none> and password file <none>
I20260927 22:43:54.813997 16404 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:54.814100 16404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:54.819343 16404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:43:54.822244 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:54.823748 16404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:43:54.823884 16404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "2eefe8f5b8de448e85872deddfdf7858"
format_stamp: "Formatted at 2026-09-27 22:43:48 on dist-test-slave-pp80"
server_key: "80cb2fd6a62129b15095fab81d5c1950"
server_key_iv: "b7830771b4d15ca20e0190d5fbdd0cf4"
server_key_version: "encryptionkey@0"
I20260927 22:43:54.824968 16404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:54.848158 16404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:54.852439 16404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:54.852669 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:54.853660 16404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:54.855028 16404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:54.855118 16404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:54.855188 16404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:54.855234 16404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:54.888926 16404 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:42735
I20260927 22:43:54.889015 16531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:42735 every 8 connection(s)
I20260927 22:43:54.890422 16404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:43:54.895258 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:54.893435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:49017)
0927 22:43:54.894163 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:54.894400 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":272,"threads_started":1}
W20260927 22:43:54.895954 16532 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:42459 (0 consecutive failures): Network error: Failed to ping master at 127.1.53.126:42459: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
I20260927 22:43:54.896855 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:54.896524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:54179)
0927 22:43:54.896674 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:54.896753 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:43:54.897527 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16404
I20260927 22:43:54.897787 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:54.897419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:57549)
0927 22:43:54.897556 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:54.897661 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:43:54.898085 16532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:43:54.898290 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:39009
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:42459
--webserver_port=36989
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:43:55.230862 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:43:55.231158 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:43:55.231252 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:43:55.238590 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:43:55.238708 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:43:55.238822 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:43:55.248611 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:39009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=36989
--tserver_master_addrs=127.1.53.126:42459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:43:55.249886 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:43:55.251173 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:43:55.265087 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:55.265105 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:43:55.265754 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:43:55.265905 16535 server_base.cc:1034] running on GCE node
I20260927 22:43:55.266569 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:43:55.267544 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:43:55.268757 16535 hybrid_clock.cc:648] HybridClock initialized: now 1790549035268701 us; error 44 us; skew 500 ppm
I20260927 22:43:55.269063 16535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:43:55 dist-test-slave-pp80 krb5kdc[15085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:43:55.272714 16535 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:43:55.274231 16535 webserver.cc:460] Webserver started at http://127.1.53.67:36989/ using document root <none> and password file <none>
I20260927 22:43:55.274987 16535 fs_manager.cc:359] Metadata directory not provided
I20260927 22:43:55.275121 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:43:55.280691 16535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:43:55.283560 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:55.285108 16535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:43:55.285297 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "19a9be5563ed426bbe6daabf56fc18ca"
format_stamp: "Formatted at 2026-09-27 22:43:49 on dist-test-slave-pp80"
server_key: "713fdd8fb027b7db3c263da878226413"
server_key_iv: "2ca1bfab6d0c905daf03a0e5bd05745e"
server_key_version: "encryptionkey@0"
I20260927 22:43:55.286331 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:43:55.304212 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:43:55.308446 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:43:55.308713 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:43:55.309739 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:43:55.311219 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:43:55.311342 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:55.311478 16535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:43:55.311571 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:43:55.345773 16535 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:39009
I20260927 22:43:55.345884 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:39009 every 8 connection(s)
I20260927 22:43:55.347251 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:43:55.351332 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16535
I20260927 22:43:55.352384 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:55.350273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:36841)
0927 22:43:55.351025 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:55.351244 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":304,"threads_started":1}
W20260927 22:43:55.353238 16663 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:42459 (0 consecutive failures): Network error: Failed to ping master at 127.1.53.126:42459: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
I20260927 22:43:55.354164 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:55.353803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:56539)
0927 22:43:55.353984 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:55.354056 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:43:55.355064 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:55.354743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:41173)
0927 22:43:55.354850 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:55.354949 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 22:43:55.355316 16663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:43:55.449213 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:55.448439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:53453)
0927 22:43:55.448908 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:55.448998 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":219,"threads_started":1}
I20260927 22:43:55.899676 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:55.898819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:40675)
0927 22:43:55.899315 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:55.899417 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260927 22:43:56.064428 16669 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:56.064734 16669 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:56.066448 16669 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
Sep 27 22:43:56 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
Sep 27 22:43:56 dist-test-slave-pp80 krb5kdc[15085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:43:56.090777 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.066729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:39009 (local address 127.1.53.65:53097)
0927 22:43:56.067463 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:56.067509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:56.068273 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:56.071866 (+ 3593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:56.071876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:56.072849 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:56.073585 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:56.073604 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.075178 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.075185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:56.076603 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:56.076619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.077664 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.077670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:56.077904 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:56.078713 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:56.078738 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:56.081422 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:56.086047 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:56.086055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:56.086066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:56.086782 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:56.087300 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:56.087304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:56.087306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:56.087556 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:56.088713 (+ 1157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:56.088721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:56.089194 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 22:43:56.089513 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:43:56.090201 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":270,"spinlock_wait_cycles":43648,"thread_start_us":751,"threads_started":1}
I20260927 22:43:56.090844 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.067181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:46241 (local address 127.1.53.66:42735)
0927 22:43:56.069125 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:56.069135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:56.069170 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:43:56.071006 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:56.071016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:56.071394 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:56.071685 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:56.073777 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.074937 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.076781 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.077387 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.077511 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:56.082148 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:56.082156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:56.082172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:56.082219 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:56.086286 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:56.087182 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:56.087191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:56.087197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:56.087388 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:56.087852 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:56.087858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:56.087862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:56.088532 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:56.088736 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:56.090075 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0927 22:43:56.090432 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1781,"thread_start_us":1603,"threads_started":1}
I20260927 22:43:56.090865 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.066933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:53097 (local address 127.1.53.67:39009)
0927 22:43:56.069104 (+ 2171us) server_negotiation.cc:206] Beginning negotiation
0927 22:43:56.069115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:43:56.069148 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:43:56.071010 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:43:56.071020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:43:56.071417 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:43:56.071746 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:43:56.073776 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.074979 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.076742 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.077454 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.077648 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:56.081752 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:43:56.081762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:43:56.081780 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:43:56.081832 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:43:56.085881 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:56.086957 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:56.086964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:56.086970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:56.087182 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:43:56.087706 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:43:56.087710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:43:56.087713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:43:56.088532 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:43:56.088736 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:43:56.089999 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0927 22:43:56.090432 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":1865,"threads_started":1}
I20260927 22:43:56.090799 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.066769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:42735 (local address 127.1.53.65:46241)
0927 22:43:56.067809 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:56.067832 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:43:56.068271 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:43:56.071870 (+ 3599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:43:56.071878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:43:56.072852 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:43:56.073586 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:56.073605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.075152 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.075159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:56.076621 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:43:56.076634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:43:56.077588 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:43:56.077596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:43:56.077912 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:43:56.078792 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:43:56.078810 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:43:56.081875 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:43:56.086432 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:56.086438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:56.086442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:56.087037 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:56.087497 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:43:56.087502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:43:56.087506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:43:56.087716 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:43:56.088713 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:43:56.088721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:43:56.089194 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 22:43:56.089513 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:43:56.090328 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":404,"spinlock_wait_cycles":58624,"thread_start_us":347,"threads_started":1}
W20260927 22:43:56.094290 16286 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
W20260927 22:43:56.094517 16288 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:43:56.094689 16288 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f4e1228bce04d2ebd66955e9877b58a; no voters: 19a9be5563ed426bbe6daabf56fc18ca, 2eefe8f5b8de448e85872deddfdf7858
I20260927 22:43:56.095098 16669 raft_consensus.cc:2726] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:43:56.356796 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.355844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:47497)
0927 22:43:56.356412 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:56.356546 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":269,"threads_started":1}
I20260927 22:43:56.450573 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.450107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:55291)
0927 22:43:56.450308 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:56.450429 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:43:56.901585 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:56.900814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:42375)
0927 22:43:56.901325 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:56.901406 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":300,"threads_started":1}
I20260927 22:43:57.358304 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:57.357650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:49167)
0927 22:43:57.358076 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:57.358157 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":227,"threads_started":1}
I20260927 22:43:57.452265 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:57.451482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:32813)
0927 22:43:57.451986 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:57.452060 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
I20260927 22:43:57.903375 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:57.902517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:54683)
0927 22:43:57.903117 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:57.903200 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":313,"threads_started":1}
I20260927 22:43:58.293622 16679 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:43:58.293798 16679 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:43:58.294536 16679 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
W20260927 22:43:58.295688 16288 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
W20260927 22:43:58.295866 16286 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:43:58.296023 16286 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f4e1228bce04d2ebd66955e9877b58a; no voters: 19a9be5563ed426bbe6daabf56fc18ca, 2eefe8f5b8de448e85872deddfdf7858
I20260927 22:43:58.296289 16679 raft_consensus.cc:2726] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:43:58.360093 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:58.359341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:43159)
0927 22:43:58.359794 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:58.359879 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260927 22:43:58.453918 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:58.453165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:39765)
0927 22:43:58.453611 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:58.453705 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
I20260927 22:43:58.905186 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:58.904464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:59923)
0927 22:43:58.904910 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:58.904992 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":215,"threads_started":1}
I20260927 22:43:59.361873 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:59.361130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:58407)
0927 22:43:59.361607 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:59.361690 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":234,"threads_started":1}
I20260927 22:43:59.455731 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:59.454926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:55285)
0927 22:43:59.455444 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:59.455545 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":239,"threads_started":1}
I20260927 22:43:59.906920 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:43:59.906217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:34277)
0927 22:43:59.906673 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:43:59.906755 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260927 22:44:00.363462 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:00.362790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:36113)
0927 22:44:00.363233 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:00.363309 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260927 22:44:00.457446 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:00.456706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:44657)
0927 22:44:00.457178 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:00.457264 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":252,"threads_started":1}
I20260927 22:44:00.908713 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:00.907851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:53387)
0927 22:44:00.908410 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:00.908508 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":273,"threads_started":1}
I20260927 22:44:01.079840 16689 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:44:01.080083 16689 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:44:01.080847 16689 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
W20260927 22:44:01.081913 16286 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
W20260927 22:44:01.082122 16288 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:44:01.082274 16288 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f4e1228bce04d2ebd66955e9877b58a; no voters: 19a9be5563ed426bbe6daabf56fc18ca, 2eefe8f5b8de448e85872deddfdf7858
I20260927 22:44:01.082553 16689 raft_consensus.cc:2726] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:44:01.365159 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:01.364395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:41767)
0927 22:44:01.364860 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:01.364972 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20260927 22:44:01.459136 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:01.458338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:58251)
0927 22:44:01.458815 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:01.458927 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260927 22:44:01.910531 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:01.909734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:59293)
0927 22:44:01.910260 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:01.910351 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":231,"threads_started":1}
I20260927 22:44:02.367029 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:02.366136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:41323)
0927 22:44:02.366703 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:02.366817 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":255,"threads_started":1}
I20260927 22:44:02.461309 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:02.460166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:58543)
0927 22:44:02.460901 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:02.461059 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":339,"threads_started":1}
I20260927 22:44:02.912174 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:02.911491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:54877)
0927 22:44:02.911896 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:02.911975 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":212,"threads_started":1}
I20260927 22:44:03.368707 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:03.367956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:48761)
0927 22:44:03.368437 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:03.368517 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":224,"threads_started":1}
I20260927 22:44:03.463094 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:03.462231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:39537)
0927 22:44:03.462780 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:03.462866 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":307,"threads_started":1}
I20260927 22:44:03.913993 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:03.913188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:50573)
0927 22:44:03.913718 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:03.913828 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
I20260927 22:44:04.370486 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:04.369676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:39149)
0927 22:44:04.370227 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:04.370327 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260927 22:44:04.464869 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:04.464030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:51351)
0927 22:44:04.464573 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:04.464676 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":319,"threads_started":1}
I20260927 22:44:04.761528 16701 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:44:04.761726 16701 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:44:04.762490 16701 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
W20260927 22:44:04.763334 16286 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
W20260927 22:44:04.763520 16288 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:44:04.763690 16288 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f4e1228bce04d2ebd66955e9877b58a; no voters: 19a9be5563ed426bbe6daabf56fc18ca, 2eefe8f5b8de448e85872deddfdf7858
I20260927 22:44:04.763952 16701 raft_consensus.cc:2726] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:44:04.915795 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:04.915018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:45249)
0927 22:44:04.915522 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:04.915604 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260927 22:44:05.372568 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:05.371605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:47835)
0927 22:44:05.372188 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:05.372333 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":370,"threads_started":1}
I20260927 22:44:05.466557 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:05.465817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:35241)
0927 22:44:05.466267 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:05.466357 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260927 22:44:05.917688 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:05.916820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:49209)
0927 22:44:05.917420 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:05.917516 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":352,"threads_started":1}
I20260927 22:44:06.374250 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:06.373471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:57153)
0927 22:44:06.373971 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:06.374077 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260927 22:44:06.468412 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:06.467614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:58973)
0927 22:44:06.468078 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:06.468178 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260927 22:44:06.919358 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:06.918618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:33193)
0927 22:44:06.919105 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:06.919185 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260927 22:44:07.376124 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:07.375336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:44055)
0927 22:44:07.375843 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:07.375949 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":270,"threads_started":1}
I20260927 22:44:07.470118 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:07.469374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:48947)
0927 22:44:07.469855 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:07.469938 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260927 22:44:07.921093 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:07.920346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:49877)
0927 22:44:07.920818 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:07.920917 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":248,"threads_started":1}
I20260927 22:44:08.165427 16712 raft_consensus.cc:482] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:44:08.165655 16712 raft_consensus.cc:504] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a9be5563ed426bbe6daabf56fc18ca" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 39009 } } peers { permanent_uuid: "2eefe8f5b8de448e85872deddfdf7858" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 42735 } } peers { permanent_uuid: "2f4e1228bce04d2ebd66955e9877b58a" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38745 } }
I20260927 22:44:08.166378 16712 leader_election.cc:288] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009), 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735)
W20260927 22:44:08.167295 16288 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2eefe8f5b8de448e85872deddfdf7858 (127.1.53.66:42735): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
W20260927 22:44:08.167486 16286 leader_election.cc:341] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19a9be5563ed426bbe6daabf56fc18ca (127.1.53.67:39009): Not found: Tablet not found: 6b0ad4a7b9ef4444ba45178d7cb3346a
I20260927 22:44:08.167662 16286 leader_election.cc:302] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f4e1228bce04d2ebd66955e9877b58a; no voters: 19a9be5563ed426bbe6daabf56fc18ca, 2eefe8f5b8de448e85872deddfdf7858
I20260927 22:44:08.167965 16712 raft_consensus.cc:2726] T 6b0ad4a7b9ef4444ba45178d7cb3346a P 2f4e1228bce04d2ebd66955e9877b58a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:44:08.378005 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:08.377248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:47757)
0927 22:44:08.377736 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:08.377831 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260927 22:44:08.471956 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:08.471063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:51151)
0927 22:44:08.471581 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:08.471697 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
I20260927 22:44:08.922837 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:08.922045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:37769)
0927 22:44:08.922525 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:08.922618 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260927 22:44:09.379755 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:09.379014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:33735)
0927 22:44:09.379466 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:09.379561 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":231,"threads_started":1}
I20260927 22:44:09.473973 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:09.473296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.65:40257)
0927 22:44:09.473713 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:09.473802 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":235,"threads_started":1}
I20260927 22:44:09.925061 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:09.924112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.66:36721)
0927 22:44:09.924779 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:09.924892 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":361,"threads_started":1}
I20260927 22:44:10.353097 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 16235
I20260927 22:44:10.377194 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:10.354351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.1.53.65:38745)
0927 22:44:10.355928 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:10.355945 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:10.355984 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:44:10.356046 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:10.356052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:10.356423 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:10.356612 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:10.358335 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:10.359344 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:10.360735 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:10.361367 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:10.361486 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:10.371077 (+ 9591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:10.371094 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:10.371109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:10.371156 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:10.374202 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:10.374932 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:10.374940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:10.374942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:10.375103 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:10.375559 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:10.375563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:10.375565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:10.375990 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:10.376346 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:10.376586 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:44:10.376803 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":1265,"threads_started":1}
I20260927 22:44:10.378758 16395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:10.382036 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:10.381147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42459 (local address 127.1.53.67:38387)
0927 22:44:10.381669 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:10.381838 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:42459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260927 22:44:10.470920 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 16235
I20260927 22:44:10.489523 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 16404
I20260927 22:44:10.504388 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:10.490452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.1.53.66:42735)
0927 22:44:10.491000 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:10.491005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:10.491022 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:44:10.491105 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:10.491109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:10.491170 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:10.491262 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:10.492792 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:10.493663 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:10.494934 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:10.495358 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:10.495457 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:10.498239 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:10.498244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:10.498248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:10.498287 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:10.501566 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:10.502296 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:10.502301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:10.502303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:10.502486 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:10.502935 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:10.502940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:10.502942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:10.503239 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:10.503482 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:10.503795 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:44:10.503982 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":330,"threads_started":1}
I20260927 22:44:10.505179 16527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:10.590406 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 16404
I20260927 22:44:10.609010 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 16535
I20260927 22:44:10.622929 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:10.609862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.1.53.67:39009)
0927 22:44:10.610440 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:10.610445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:10.610463 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:44:10.610520 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:10.610523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:10.610617 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:10.610740 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:10.612097 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:10.612949 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:10.614196 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:10.614616 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:10.614754 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:10.617409 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:10.617414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:10.617417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:10.617448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:10.620136 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:10.620858 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:10.620864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:10.620868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:10.621018 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:10.621434 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:10.621438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:10.621440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:10.621719 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:10.622016 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:10.622302 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 22:44:10.622541 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":296,"threads_started":1}
I20260927 22:44:10.623761 16658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:10.708983 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 16535
2026-09-27T22:44:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24904 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:44:10 dist-test-slave-pp80 krb5kdc[16728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:44:10 dist-test-slave-pp80 krb5kdc[16728](info): set up 2 sockets
Sep 27 22:44:10 dist-test-slave-pp80 krb5kdc[16728](info): commencing operation
krb5kdc: starting...
W20260927 22:44:11.820963 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:44:11 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:44:12.068360 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46099
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:12.408732 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:12.409029 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:12.409209 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:12.416723 16741 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:44:12.416831 16741 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:44:12.416903 16741 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:44:12.417004 16741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:44:12.417096 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:12.417218 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:12.417281 16741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:44:12.417377 16741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:44:12.427685 16741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:12.429100 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:12.430519 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:12.442535 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:12.442587 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:12.443331 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:12.443356 16741 server_base.cc:1034] running on GCE node
I20260927 22:44:12.444289 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:12.445806 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:12.447047 16741 hybrid_clock.cc:648] HybridClock initialized: now 1790549052446990 us; error 29 us; skew 500 ppm
I20260927 22:44:12.447412 16741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:12 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549052, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:12.450712 16741 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:44:12.452375 16741 webserver.cc:460] Webserver started at http://127.1.53.126:40781/ using document root <none> and password file <none>
I20260927 22:44:12.453069 16741 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:12.453152 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:12.453418 16741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:44:12.456368 16741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "6a1bb577bb1a42418bc6bdb967169edc"
format_stamp: "Formatted at 2026-09-27 22:44:12 on dist-test-slave-pp80"
server_key: "f6481053abd69b34f0f7cd533b7928a6"
server_key_iv: "bfb031a6d8ba4884fb2c515a7579b256"
server_key_version: "encryptionkey@0"
I20260927 22:44:12.457149 16741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "6a1bb577bb1a42418bc6bdb967169edc"
format_stamp: "Formatted at 2026-09-27 22:44:12 on dist-test-slave-pp80"
server_key: "f6481053abd69b34f0f7cd533b7928a6"
server_key_iv: "bfb031a6d8ba4884fb2c515a7579b256"
server_key_version: "encryptionkey@0"
I20260927 22:44:12.462469 16741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:44:12.466138 16756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:12.467721 16741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:44:12.467900 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "6a1bb577bb1a42418bc6bdb967169edc"
format_stamp: "Formatted at 2026-09-27 22:44:12 on dist-test-slave-pp80"
server_key: "f6481053abd69b34f0f7cd533b7928a6"
server_key_iv: "bfb031a6d8ba4884fb2c515a7579b256"
server_key_version: "encryptionkey@0"
I20260927 22:44:12.468128 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:12.484632 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:12.488993 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:12.489287 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:12.508728 16741 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:46099
I20260927 22:44:12.508750 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:46099 every 8 connection(s)
I20260927 22:44:12.510133 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:44:12.512380 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16741
I20260927 22:44:12.512626 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:44:12.513449 1237 external_mini_cluster.cc:1428] Setting key dc623a7981fcb11edadde7791153028c
I20260927 22:44:12.514727 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:44:12 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:12.529974 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap starting.
I20260927 22:44:12.534291 16808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:12.535403 16808 log.cc:824] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:12.537989 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:12.515469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.1.53.126:46099)
0927 22:44:12.516336 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:12.516347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:12.516389 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:44:12.517981 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:12.518002 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:12.518391 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:12.518719 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:12.520698 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:12.521996 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:12.523937 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:12.524670 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:12.524799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:12.529760 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:12.529771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:12.529790 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:12.529843 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:12.533812 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:12.534575 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:12.534587 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:12.534593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:12.534774 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:12.535169 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:12.535176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:12.535184 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:12.535789 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:12.536083 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:12.536595 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 22:44:12.537010 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":353,"threads_started":1}
I20260927 22:44:12.538534 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: No bootstrap required, opened a new log
I20260927 22:44:12.546622 16808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:12.547104 16808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:12.547230 16808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bb577bb1a42418bc6bdb967169edc, State: Initialized, Role: FOLLOWER
I20260927 22:44:12.547834 16808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:12.548043 16808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:44:12.548144 16808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:44:12.548348 16808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:12.550452 16808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:12.550940 16808 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1bb577bb1a42418bc6bdb967169edc; no voters:
I20260927 22:44:12.551565 16808 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:44:12.551712 16813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:44:12.552455 16813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 LEADER]: Becoming Leader. State: Replica: 6a1bb577bb1a42418bc6bdb967169edc, State: Running, Role: LEADER
I20260927 22:44:12.552954 16813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:12.553776 16808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:44:12.555805 16815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1bb577bb1a42418bc6bdb967169edc. Latest consensus state: current_term: 1 leader_uuid: "6a1bb577bb1a42418bc6bdb967169edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } } }
I20260927 22:44:12.556167 16815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: This master's current role is: LEADER
I20260927 22:44:12.556756 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1bb577bb1a42418bc6bdb967169edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } } }
I20260927 22:44:12.557195 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: This master's current role is: LEADER
I20260927 22:44:12.558573 16818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:44:12.563885 16818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:44:12.574471 16818 catalog_manager.cc:1347] Generated new cluster ID: d5518858836d4aec9efaa3ee7a09fdfe
I20260927 22:44:12.574615 16818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:44:12.589299 16818 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:44:12.590682 16818 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:44:12.597363 16818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Generated new TSK 0
I20260927 22:44:12.598356 16818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:44:12.664950 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46099
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:13.023167 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:13.023450 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:13.023614 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:13.030769 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:13.030874 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:13.031059 16834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:44:13.041353 16834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:13.042668 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:13.043998 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:13.057657 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:13.057950 16839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:13.057894 16840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:13.059126 16834 server_base.cc:1034] running on GCE node
I20260927 22:44:13.059885 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:13.060982 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:13.062196 16834 hybrid_clock.cc:648] HybridClock initialized: now 1790549053062151 us; error 33 us; skew 500 ppm
I20260927 22:44:13.062551 16834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:13 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:13.066159 16834 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:44:13.067778 16834 webserver.cc:460] Webserver started at http://127.1.53.65:44883/ using document root <none> and password file <none>
I20260927 22:44:13.068686 16834 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:13.068856 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:13.069226 16834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:44:13.072642 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "6465cd858a7d439e978b937acf518396"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "2410205ac95c3b41239a2214828b59e5"
server_key_iv: "89f11d8196395033203e941b2a88372e"
server_key_version: "encryptionkey@0"
I20260927 22:44:13.073508 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "6465cd858a7d439e978b937acf518396"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "2410205ac95c3b41239a2214828b59e5"
server_key_iv: "89f11d8196395033203e941b2a88372e"
server_key_version: "encryptionkey@0"
I20260927 22:44:13.079319 16834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:44:13.083168 16849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:13.084739 16834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:44:13.084944 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "6465cd858a7d439e978b937acf518396"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "2410205ac95c3b41239a2214828b59e5"
server_key_iv: "89f11d8196395033203e941b2a88372e"
server_key_version: "encryptionkey@0"
I20260927 22:44:13.085181 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:13.111618 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:13.116683 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:13.117008 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:13.118091 16834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:13.119535 16834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:13.119660 16834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:13.119834 16834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:13.119915 16834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:13.156719 16834 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:44225
I20260927 22:44:13.156780 16961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:44225 every 8 connection(s)
I20260927 22:44:13.158123 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:44:13.163892 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16834
I20260927 22:44:13.164121 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:44:13.164821 1237 external_mini_cluster.cc:1428] Setting key 0e3a0a70e376116b09b0083ea8a173cf
Sep 27 22:44:13 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:13.181384 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:13.160941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:40705 (local address 127.1.53.126:46099)
0927 22:44:13.161380 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:13.161386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:13.162469 (+ 1083us) server_negotiation.cc:407] Connection header received
0927 22:44:13.164733 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:13.164740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:13.164844 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:13.164984 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:13.167197 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.168310 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.169970 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.170675 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.170818 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:13.174736 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:13.174746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:13.174752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:13.174801 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:13.177890 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:13.178894 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:13.178898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:13.178900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:13.179022 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:13.179512 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:13.179515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:13.179517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:13.179782 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:13.179902 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:13.180915 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0927 22:44:13.181121 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260927 22:44:13.183032 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:13.161551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:40705)
0927 22:44:13.162284 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:13.162331 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:13.164431 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:13.165262 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:13.165270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:13.166093 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:44:13.166983 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:13.167014 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.168461 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.168465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:13.169813 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:13.169826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.170898 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.170902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:13.171136 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:13.171945 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:44:13.171970 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:44:13.174480 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:44:13.178050 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:13.178063 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:13.178087 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:13.178770 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:13.179137 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:13.179145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:13.179151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:13.179407 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:13.179925 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:44:13.179936 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:44:13.180459 (+ 523us) client_negotiation.cc:769] Sending connection context
0927 22:44:13.180816 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 22:44:13.181448 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":293,"threads_started":1}
I20260927 22:44:13.184693 16962 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46099
I20260927 22:44:13.185014 16962 heartbeater.cc:464] Registering TS with master...
I20260927 22:44:13.185762 16962 heartbeater.cc:511] Master 127.1.53.126:46099 requested a full tablet report, sending...
I20260927 22:44:13.187999 16773 ts_manager.cc:194] Registered new tserver with Master: 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225)
I20260927 22:44:13.189706 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:40705
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:44:13.224078 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46099
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:13.569569 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:13.569958 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:13.570139 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:13.578868 16968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:13.579067 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:13.579275 16968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:44:13.591061 16968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:13.592628 16968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:13.594141 16968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:13.608850 16973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:13.608815 16976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:13.608891 16974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:13.610275 16968 server_base.cc:1034] running on GCE node
I20260927 22:44:13.611047 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:13.612197 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:13.613478 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790549053613412 us; error 58 us; skew 500 ppm
I20260927 22:44:13.613829 16968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:13 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:13.617234 16968 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:44:13.618888 16968 webserver.cc:460] Webserver started at http://127.1.53.66:40983/ using document root <none> and password file <none>
I20260927 22:44:13.619586 16968 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:13.619688 16968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:13.619941 16968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:44:13.623020 16968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "026402c65d41406ea13c1f24394fbd0e"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "01c26576efe4a250b697474f72369897"
server_key_iv: "c8bd54001a14647646784f6980e06239"
server_key_version: "encryptionkey@0"
I20260927 22:44:13.623921 16968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "026402c65d41406ea13c1f24394fbd0e"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "01c26576efe4a250b697474f72369897"
server_key_iv: "c8bd54001a14647646784f6980e06239"
server_key_version: "encryptionkey@0"
I20260927 22:44:13.629477 16968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:44:13.633162 16983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:13.634559 16968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:44:13.634701 16968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "026402c65d41406ea13c1f24394fbd0e"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "01c26576efe4a250b697474f72369897"
server_key_iv: "c8bd54001a14647646784f6980e06239"
server_key_version: "encryptionkey@0"
I20260927 22:44:13.634869 16968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:13.651211 16968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:13.655815 16968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:13.656064 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:13.657119 16968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:13.658480 16968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:13.658577 16968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:13.658715 16968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:13.658794 16968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:13.693909 16968 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:33679
I20260927 22:44:13.693998 17095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:33679 every 8 connection(s)
I20260927 22:44:13.695233 16968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:44:13.698639 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 16968
I20260927 22:44:13.698828 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:44:13.699326 1237 external_mini_cluster.cc:1428] Setting key 2be84f5cc5ce887a9cbd6d65581cb2bd
Sep 27 22:44:13 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549053, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:13.720136 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:13.697952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:55747 (local address 127.1.53.126:46099)
0927 22:44:13.698511 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:13.698518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:13.699614 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 22:44:13.701856 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:13.701860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:13.701930 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:13.702035 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:13.704234 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.705093 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.707704 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.708128 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.708227 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:13.712702 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:13.712712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:13.712720 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:13.712772 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:13.716497 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:13.717301 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:13.717309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:13.717316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:13.717484 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:13.717924 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:13.717932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:13.717938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:13.718352 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:13.718507 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:13.719350 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 22:44:13.719612 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":269,"threads_started":1}
I20260927 22:44:13.720970 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:13.698558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:55747)
0927 22:44:13.699429 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:13.699484 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:13.701542 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:13.702327 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:13.702342 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:13.703179 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:44:13.704040 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:13.704066 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.705261 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.705269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:13.707532 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:13.707549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:13.708386 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:13.708395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:13.708744 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:13.709523 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:44:13.709559 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:44:13.712376 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:44:13.716673 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:13.716681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:13.716696 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:13.717164 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:13.717607 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:13.717610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:13.717612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:13.717803 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:13.718541 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:44:13.718549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:44:13.718932 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 22:44:13.719220 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 22:44:13.719674 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":385,"threads_started":1}
I20260927 22:44:13.722774 17096 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46099
I20260927 22:44:13.723147 17096 heartbeater.cc:464] Registering TS with master...
I20260927 22:44:13.724164 17096 heartbeater.cc:511] Master 127.1.53.126:46099 requested a full tablet report, sending...
I20260927 22:44:13.725843 16773 ts_manager.cc:194] Registered new tserver with Master: 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
I20260927 22:44:13.726801 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:55747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:44:13.760324 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46099
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:14.089298 17102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:14.089586 17102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:14.089685 17102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:14.097013 17102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:14.097143 17102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:14.097263 17102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:44:14.107149 17102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:14.108501 17102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:14.109762 17102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:14.123461 17107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:14.123489 17110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:14.123490 17108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:14.124359 17102 server_base.cc:1034] running on GCE node
I20260927 22:44:14.124980 17102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:14.126020 17102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:14.127274 17102 hybrid_clock.cc:648] HybridClock initialized: now 1790549054127196 us; error 71 us; skew 500 ppm
I20260927 22:44:14.127619 17102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:14 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:14.130860 17102 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:44:14.132426 17102 webserver.cc:460] Webserver started at http://127.1.53.67:46837/ using document root <none> and password file <none>
I20260927 22:44:14.133162 17102 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:14.133313 17102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:14.133641 17102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:44:14.136641 17102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "d1fa1097793b47078eb58e42f4345906"
format_stamp: "Formatted at 2026-09-27 22:44:14 on dist-test-slave-pp80"
server_key: "f40e6c5a830d139f69fd716e3627bd09"
server_key_iv: "dcc864295818c6006e106fc84800d95e"
server_key_version: "encryptionkey@0"
I20260927 22:44:14.137405 17102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "d1fa1097793b47078eb58e42f4345906"
format_stamp: "Formatted at 2026-09-27 22:44:14 on dist-test-slave-pp80"
server_key: "f40e6c5a830d139f69fd716e3627bd09"
server_key_iv: "dcc864295818c6006e106fc84800d95e"
server_key_version: "encryptionkey@0"
I20260927 22:44:14.142795 17102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:44:14.146422 17117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:14.147832 17102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:44:14.148027 17102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "d1fa1097793b47078eb58e42f4345906"
format_stamp: "Formatted at 2026-09-27 22:44:14 on dist-test-slave-pp80"
server_key: "f40e6c5a830d139f69fd716e3627bd09"
server_key_iv: "dcc864295818c6006e106fc84800d95e"
server_key_version: "encryptionkey@0"
I20260927 22:44:14.148252 17102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:14.159931 17102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:14.164217 17102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:14.164565 17102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:14.165532 17102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:14.167050 17102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:14.167160 17102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:14.167290 17102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:14.167379 17102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:14.193348 16962 heartbeater.cc:503] Master 127.1.53.126:46099 was elected leader, sending a full tablet report...
I20260927 22:44:14.202793 17102 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:44139
I20260927 22:44:14.202847 17229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:44139 every 8 connection(s)
I20260927 22:44:14.204149 17102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:44:14.204649 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17102
I20260927 22:44:14.204870 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:44:14.205495 1237 external_mini_cluster.cc:1428] Setting key de244670a92739b543d75b441c0d9723
Sep 27 22:44:14 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:14.226227 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.207383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:59747 (local address 127.1.53.126:46099)
0927 22:44:14.207633 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.207639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.209202 (+ 1563us) server_negotiation.cc:407] Connection header received
0927 22:44:14.211241 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.211245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.211324 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.211412 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:14.213224 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.213992 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.215653 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.216055 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.216154 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.219703 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:14.219709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:14.219712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:14.219750 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:14.223085 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.223889 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.223895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.223899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.224043 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.224628 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.224632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.224634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.224942 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:14.225056 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.225785 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.225980 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:44:14.227253 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.207990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:59747)
0927 22:44:14.209031 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:14.209082 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:14.210970 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:14.211698 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:14.211706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:14.212375 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:44:14.213035 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.213055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.214140 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.214145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.215516 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.215528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.216330 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.216338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.216565 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.217271 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:44:14.217294 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:44:14.219463 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:44:14.223238 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:14.223245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:14.223258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:14.223733 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:14.224175 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:14.224179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:14.224182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:14.224504 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:14.225063 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:44:14.225070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:44:14.225398 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 22:44:14.225649 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 22:44:14.226089 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":529,"threads_started":1}
I20260927 22:44:14.228735 17230 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46099
I20260927 22:44:14.229105 17230 heartbeater.cc:464] Registering TS with master...
I20260927 22:44:14.229866 17230 heartbeater.cc:511] Master 127.1.53.126:46099 requested a full tablet report, sending...
I20260927 22:44:14.231341 16773 ts_manager.cc:194] Registered new tserver with Master: d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139)
I20260927 22:44:14.232295 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:59747
I20260927 22:44:14.234303 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:44:14.252686 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.239122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.1.53.126:46099)
0927 22:44:14.239345 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.239355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.239743 (+ 388us) server_negotiation.cc:407] Connection header received
0927 22:44:14.240019 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.240023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.240100 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.240205 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:14.241620 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.242314 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.243754 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.244167 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.244324 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.246786 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:14.246791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:14.246794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:14.246824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:14.249652 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.250457 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.250463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.250466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.250603 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.251024 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.251029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.251032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.251451 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:14.251726 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.252067 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.252330 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:44:14.258472 16772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:44:14.267802 16772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:44:14.293673 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.281070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:40614)
0927 22:44:14.281705 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:14.281760 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:14.281951 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:14.283014 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:14.283026 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:14.283068 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:14.283551 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.283566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.286131 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.286139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.289337 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.289359 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.292273 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.292283 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.292613 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.292687 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 22:44:14.292854 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:44:14.293126 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I20260927 22:44:14.295006 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.281387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.1.53.65:44225)
0927 22:44:14.282037 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.282048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.282076 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:44:14.282177 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.282186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.282473 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.282693 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:14.283788 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.285914 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.289589 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.291616 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.291784 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.292092 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.294298 (+ 2206us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.294547 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":305,"threads_started":1}
I20260927 22:44:14.302366 16897 tablet_service.cc:1467] Processing CreateTablet for tablet ee3c504bc9dd45de90490343e56d30bc (DEFAULT_TABLE table=test-table [id=08da5e2a0c3948a396081333515ad837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:44:14.303249 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.283374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.1.53.66:33679)
0927 22:44:14.295921 (+ 12547us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.295934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.295975 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:44:14.296102 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.296114 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.296474 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.297514 (+ 1040us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:14.297780 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.299258 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.301109 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.302279 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.302419 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.302618 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.302769 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.302925 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12289,"thread_start_us":13296,"threads_started":1}
I20260927 22:44:14.303290 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.283095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33679 (local address 127.0.0.1:39294)
0927 22:44:14.293946 (+ 10851us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:14.293965 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:14.294085 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:14.296985 (+ 2900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:14.296994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:14.297032 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:14.297287 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.297302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.299416 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.299420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.300953 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.300965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.302428 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.302434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.302602 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.302637 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:44:14.302727 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:44:14.302862 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10761,"mutex_wait_us":78,"thread_start_us":14627,"threads_started":1}
I20260927 22:44:14.304467 16897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3c504bc9dd45de90490343e56d30bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:44:14.304708 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.282152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:44139 (local address 127.0.0.1:40376)
0927 22:44:14.282665 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:14.282681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:14.282787 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:14.297298 (+ 14511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:14.297303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:14.297329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:14.297874 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.297889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.299585 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.299590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.301737 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.301751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.303929 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.303938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.304112 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.304144 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:44:14.304212 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 22:44:14.304375 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":280,"threads_started":1}
I20260927 22:44:14.305785 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.282476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.1.53.67:44139)
0927 22:44:14.296812 (+ 14336us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.296819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.296845 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:44:14.296925 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.296931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.297097 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.297315 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:14.298098 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.299421 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.301923 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.303693 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.305012 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.305225 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.305299 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.305457 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14170,"thread_start_us":14027,"threads_started":1}
I20260927 22:44:14.306844 17031 tablet_service.cc:1467] Processing CreateTablet for tablet ee3c504bc9dd45de90490343e56d30bc (DEFAULT_TABLE table=test-table [id=08da5e2a0c3948a396081333515ad837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:44:14.308286 17031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3c504bc9dd45de90490343e56d30bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:44:14.309687 17165 tablet_service.cc:1467] Processing CreateTablet for tablet ee3c504bc9dd45de90490343e56d30bc (DEFAULT_TABLE table=test-table [id=08da5e2a0c3948a396081333515ad837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:44:14.311128 17165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3c504bc9dd45de90490343e56d30bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:44:14.316481 17248 tablet_bootstrap.cc:492] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Bootstrap starting.
I20260927 22:44:14.321041 17248 tablet_bootstrap.cc:654] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:14.322078 17248 log.cc:824] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:14.322964 17249 tablet_bootstrap.cc:492] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Bootstrap starting.
I20260927 22:44:14.325387 17251 tablet_bootstrap.cc:492] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Bootstrap starting.
I20260927 22:44:14.326462 17248 tablet_bootstrap.cc:492] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: No bootstrap required, opened a new log
I20260927 22:44:14.326747 17248 ts_tablet_manager.cc:1397] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260927 22:44:14.327666 17249 tablet_bootstrap.cc:654] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:14.328223 17251 tablet_bootstrap.cc:654] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:14.328918 17249 log.cc:824] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:14.329245 17251 log.cc:824] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:14.332002 17249 tablet_bootstrap.cc:492] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: No bootstrap required, opened a new log
I20260927 22:44:14.332019 17251 tablet_bootstrap.cc:492] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: No bootstrap required, opened a new log
I20260927 22:44:14.332307 17249 ts_tablet_manager.cc:1397] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:44:14.332305 17251 ts_tablet_manager.cc:1397] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260927 22:44:14.335716 17248 raft_consensus.cc:348] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.336114 17248 raft_consensus.cc:374] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:14.336284 17248 raft_consensus.cc:729] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6465cd858a7d439e978b937acf518396, State: Initialized, Role: FOLLOWER
I20260927 22:44:14.336946 17248 consensus_queue.cc:260] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.338874 17248 ts_tablet_manager.cc:1428] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
I20260927 22:44:14.343191 17249 raft_consensus.cc:348] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.343171 17251 raft_consensus.cc:348] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.343627 17251 raft_consensus.cc:374] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:14.343627 17249 raft_consensus.cc:374] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:14.343730 17249 raft_consensus.cc:729] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026402c65d41406ea13c1f24394fbd0e, State: Initialized, Role: FOLLOWER
I20260927 22:44:14.343730 17251 raft_consensus.cc:729] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fa1097793b47078eb58e42f4345906, State: Initialized, Role: FOLLOWER
I20260927 22:44:14.344352 17251 consensus_queue.cc:260] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.344360 17249 consensus_queue.cc:260] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.345562 17230 heartbeater.cc:503] Master 127.1.53.126:46099 was elected leader, sending a full tablet report...
I20260927 22:44:14.346066 17096 heartbeater.cc:503] Master 127.1.53.126:46099 was elected leader, sending a full tablet report...
I20260927 22:44:14.346274 17251 ts_tablet_manager.cc:1428] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 22:44:14.346581 17249 ts_tablet_manager.cc:1428] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 22:44:14.378777 17255 raft_consensus.cc:482] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:44:14.379138 17255 raft_consensus.cc:504] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.381074 17255 leader_election.cc:288] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225), d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139)
I20260927 22:44:14.388461 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.381639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:59273 (local address 127.1.53.65:44225)
0927 22:44:14.381922 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.381931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.382364 (+ 433us) server_negotiation.cc:407] Connection header received
0927 22:44:14.382623 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.382630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.382722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.382817 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:14.383563 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.384581 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.386577 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.387435 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.387562 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.387675 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.388051 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.388185 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:44:14.388532 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.382647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:59269 (local address 127.1.53.67:44139)
0927 22:44:14.382868 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.382875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.383143 (+ 268us) server_negotiation.cc:407] Connection header received
0927 22:44:14.383281 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.383286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.383373 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.383494 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:14.384086 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.385016 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.386759 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.387541 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.387668 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.387764 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.388141 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.388297 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:44:14.388602 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.381461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.1.53.66:59273)
0927 22:44:14.382259 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:14.382284 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:14.382474 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:14.382951 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:14.382955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:14.382978 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:14.383405 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.383422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.384709 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.384715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.386399 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.386411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.387674 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.387681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.387865 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.387913 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:44:14.387991 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 22:44:14.388128 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":4,"spinlock_wait_cycles":31488,"thread_start_us":399,"threads_started":1}
I20260927 22:44:14.388727 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.382304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:44139 (local address 127.1.53.66:59269)
0927 22:44:14.383029 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:14.383049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:14.383169 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:14.383521 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:14.383533 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:14.383556 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:14.383924 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.383937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.385156 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.385160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.386607 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:14.386620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.387765 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.387770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:14.387946 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.387992 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:44:14.388140 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:44:14.388282 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":41,"spinlock_wait_cycles":2816,"thread_start_us":410,"threads_started":1}
I20260927 22:44:14.389858 16917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3c504bc9dd45de90490343e56d30bc" candidate_uuid: "026402c65d41406ea13c1f24394fbd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6465cd858a7d439e978b937acf518396" is_pre_election: true
I20260927 22:44:14.389858 17185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3c504bc9dd45de90490343e56d30bc" candidate_uuid: "026402c65d41406ea13c1f24394fbd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1fa1097793b47078eb58e42f4345906" is_pre_election: true
I20260927 22:44:14.390463 17185 raft_consensus.cc:2454] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026402c65d41406ea13c1f24394fbd0e in term 0.
I20260927 22:44:14.390473 16917 raft_consensus.cc:2454] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026402c65d41406ea13c1f24394fbd0e in term 0.
I20260927 22:44:14.391137 16987 leader_election.cc:302] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906; no voters:
I20260927 22:44:14.391465 17255 raft_consensus.cc:2781] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:44:14.391635 17255 raft_consensus.cc:482] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:44:14.391746 17255 raft_consensus.cc:3037] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:14.394022 17255 raft_consensus.cc:504] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.394609 17255 leader_election.cc:288] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [CANDIDATE]: Term 1 election: Requested vote from peers 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225), d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139)
I20260927 22:44:14.395152 16917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3c504bc9dd45de90490343e56d30bc" candidate_uuid: "026402c65d41406ea13c1f24394fbd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6465cd858a7d439e978b937acf518396"
I20260927 22:44:14.395231 17185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3c504bc9dd45de90490343e56d30bc" candidate_uuid: "026402c65d41406ea13c1f24394fbd0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1fa1097793b47078eb58e42f4345906"
I20260927 22:44:14.395424 17185 raft_consensus.cc:3037] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:14.395412 16917 raft_consensus.cc:3037] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:14.398766 16917 raft_consensus.cc:2454] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026402c65d41406ea13c1f24394fbd0e in term 1.
I20260927 22:44:14.398766 17185 raft_consensus.cc:2454] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026402c65d41406ea13c1f24394fbd0e in term 1.
I20260927 22:44:14.399210 16987 leader_election.cc:302] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906; no voters:
I20260927 22:44:14.399497 17255 raft_consensus.cc:2781] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:44:14.399976 17255 raft_consensus.cc:686] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 1 LEADER]: Becoming Leader. State: Replica: 026402c65d41406ea13c1f24394fbd0e, State: Running, Role: LEADER
I20260927 22:44:14.400516 17255 consensus_queue.cc:237] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:14.405681 16772 catalog_manager.cc:5495] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 026402c65d41406ea13c1f24394fbd0e (127.1.53.66). New cstate: current_term: 1 leader_uuid: "026402c65d41406ea13c1f24394fbd0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } health_report { overall_health: HEALTHY } } }
W20260927 22:44:14.410691 16963 tablet.cc:2369] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:44:14.439983 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.433943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.1.53.66:33679)
0927 22:44:14.434137 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.434142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.434168 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:44:14.434260 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.434264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.434360 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.434515 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:44:14.434921 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.435664 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.437577 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.438126 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.438241 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.438844 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:44:14.439373 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:44:14.439546 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.439602 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.439684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:44:14.447304 17097 tablet.cc:2369] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:44:14.448868 16917 raft_consensus.cc:1261] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Refusing update from remote peer 026402c65d41406ea13c1f24394fbd0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:44:14.448865 17185 raft_consensus.cc:1261] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Refusing update from remote peer 026402c65d41406ea13c1f24394fbd0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:44:14.449913 17259 consensus_queue.cc:1035] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:44:14.450348 17255 consensus_queue.cc:1035] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:44:14.456015 17231 tablet.cc:2369] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:44:14.481101 16772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:44:14.481451 16772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:44:14.486662 16772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Sending DeleteTablet for 3 replicas of tablet ee3c504bc9dd45de90490343e56d30bc
I20260927 22:44:14.488590 17031 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3c504bc9dd45de90490343e56d30bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:44:14 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:39294
I20260927 22:44:14.488866 17165 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3c504bc9dd45de90490343e56d30bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:44:14 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:40376
I20260927 22:44:14.489585 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 16834
I20260927 22:44:14.490389 16897 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3c504bc9dd45de90490343e56d30bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:44:14 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:40614
I20260927 22:44:14.491577 17274 tablet_replica.cc:330] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: stopping tablet replica
I20260927 22:44:14.492105 17274 raft_consensus.cc:2229] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:44:14.492751 17274 raft_consensus.cc:2258] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:44:14.495250 17274 ts_tablet_manager.cc:1905] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:44:14.505121 17272 tablet_replica.cc:330] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: stopping tablet replica
I20260927 22:44:14.505959 17272 raft_consensus.cc:2229] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:44:14.506655 17272 raft_consensus.cc:2258] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:44:14.507476 17273 tablet_replica.cc:330] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: stopping tablet replica
I20260927 22:44:14.507911 17273 raft_consensus.cc:2229] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:44:14.508509 17273 raft_consensus.cc:2258] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:44:14.509598 17272 ts_tablet_manager.cc:1905] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:44:14.510761 17273 ts_tablet_manager.cc:1905] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:44:14.512344 17274 ts_tablet_manager.cc:1918] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:44:14.515491 17274 log.cc:1196] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/ee3c504bc9dd45de90490343e56d30bc
Sep 27 22:44:14 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:44:14.516125 17274 ts_tablet_manager.cc:1939] T ee3c504bc9dd45de90490343e56d30bc P 6465cd858a7d439e978b937acf518396: Deleting consensus metadata
I20260927 22:44:14.519042 17273 ts_tablet_manager.cc:1918] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:44:14.519330 17273 log.cc:1196] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/ee3c504bc9dd45de90490343e56d30bc
I20260927 22:44:14.519348 16760 catalog_manager.cc:4841] TS 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225): tablet ee3c504bc9dd45de90490343e56d30bc (table test-table [id=08da5e2a0c3948a396081333515ad837]) successfully deleted
I20260927 22:44:14.519941 17273 ts_tablet_manager.cc:1939] T ee3c504bc9dd45de90490343e56d30bc P d1fa1097793b47078eb58e42f4345906: Deleting consensus metadata
I20260927 22:44:14.522528 16760 catalog_manager.cc:4841] TS d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139): tablet ee3c504bc9dd45de90490343e56d30bc (table test-table [id=08da5e2a0c3948a396081333515ad837]) successfully deleted
I20260927 22:44:14.524770 17272 ts_tablet_manager.cc:1918] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:44:14.524966 17272 log.cc:1196] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/ee3c504bc9dd45de90490343e56d30bc
I20260927 22:44:14.525455 17272 ts_tablet_manager.cc:1939] T ee3c504bc9dd45de90490343e56d30bc P 026402c65d41406ea13c1f24394fbd0e: Deleting consensus metadata
I20260927 22:44:14.526590 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.491122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.1.53.65:44225)
0927 22:44:14.491303 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.491311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.495974 (+ 4663us) server_negotiation.cc:407] Connection header received
0927 22:44:14.496362 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.496370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.496488 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.496577 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:14.503604 (+ 7027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.504987 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.507074 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.507714 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.507818 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.519792 (+ 11974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:14.519810 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:14.519831 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:14.519877 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:14.523522 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.524302 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.524318 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.524322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.524470 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.524877 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.524883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.524886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.525305 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:14.525607 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.525970 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.526188 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:44:14.527064 16760 catalog_manager.cc:4841] TS 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679): tablet ee3c504bc9dd45de90490343e56d30bc (table test-table [id=08da5e2a0c3948a396081333515ad837]) successfully deleted
I20260927 22:44:14.527257 16957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:14.619658 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 16834
I20260927 22:44:14.644490 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 16968
Sep 27 22:44:14 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:44:14.661375 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.645921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.1.53.66:33679)
0927 22:44:14.646094 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.646098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.646114 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:44:14.646169 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.646172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.646252 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.646367 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:14.647654 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.648655 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.649956 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.650409 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.650551 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.655003 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:14.655015 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:14.655030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:14.655070 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:14.658263 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.659040 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.659048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.659050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.659229 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.659705 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.659711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.659714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.660140 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:14.660489 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.660778 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.660999 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:44:14.662216 17091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:14.776293 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 16968
I20260927 22:44:14.795529 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 17102
Sep 27 22:44:14 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:44:14.811367 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:14.796383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.1.53.67:44139)
0927 22:44:14.796582 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:14.796587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:14.796606 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:44:14.796721 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:14.796726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:14.796808 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:14.796900 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:14.798299 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.799063 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.800514 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:14.800949 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:14.801045 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:14.805696 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:14.805711 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:14.805727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:14.805763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:14.808597 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.809305 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.809312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.809314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.809447 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:14.809878 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:14.809883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:14.809885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:14.810257 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:14.810482 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:14.810784 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:44:14.810963 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:44:14.812062 17225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:14.902592 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 17102
I20260927 22:44:14.918990 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 16741
I20260927 22:44:14.920176 16803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:15.030655 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 16741
I20260927 22:44:35.053872 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46099
--webserver_interface=127.1.53.126
--webserver_port=40781
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:35.400516 17280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:35.400840 17280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:35.401005 17280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:35.408777 17280 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:44:35.408891 17280 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:44:35.408936 17280 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:44:35.409024 17280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:44:35.409121 17280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:35.409237 17280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:35.409301 17280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:44:35.409399 17280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:44:35.420680 17280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:35.422109 17280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:35.423521 17280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:35.449949 17287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:35.450253 17280 server_base.cc:1034] running on GCE node
W20260927 22:44:35.450711 17289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:35.450879 17286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:35.452073 17280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:35.454044 17280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:35.455334 17280 hybrid_clock.cc:648] HybridClock initialized: now 1790549075455254 us; error 54 us; skew 500 ppm
I20260927 22:44:35.455853 17280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:35 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:35.465008 17280 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:44:35.467258 17280 webserver.cc:460] Webserver started at http://127.1.53.126:40781/ using document root <none> and password file <none>
I20260927 22:44:35.468400 17280 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:35.468572 17280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:35.475742 17280 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 22:44:35.479409 17296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:35.481084 17280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260927 22:44:35.481300 17280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "6a1bb577bb1a42418bc6bdb967169edc"
format_stamp: "Formatted at 2026-09-27 22:44:12 on dist-test-slave-pp80"
server_key: "f6481053abd69b34f0f7cd533b7928a6"
server_key_iv: "bfb031a6d8ba4884fb2c515a7579b256"
server_key_version: "encryptionkey@0"
I20260927 22:44:35.482409 17280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:35.502351 17280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:35.507587 17280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:35.507953 17280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:35.528767 17280 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:46099
I20260927 22:44:35.528803 17347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:46099 every 8 connection(s)
I20260927 22:44:35.530510 17280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:44:35.538098 17348 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:44:35.539937 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17280
I20260927 22:44:35.540833 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:44225
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:46099
--webserver_port=44883
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:44:35.541730 17348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap starting.
I20260927 22:44:35.562781 17348 log.cc:824] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:35.575691 17348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:44:35.576385 17348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap complete.
I20260927 22:44:35.586042 17348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:35.587100 17348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bb577bb1a42418bc6bdb967169edc, State: Initialized, Role: FOLLOWER
I20260927 22:44:35.587867 17348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:35.588158 17348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:44:35.588315 17348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:44:35.588510 17348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:44:35.592540 17348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:35.593138 17348 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1bb577bb1a42418bc6bdb967169edc; no voters:
I20260927 22:44:35.593874 17348 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:44:35.594064 17352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:44:35.595403 17352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 LEADER]: Becoming Leader. State: Replica: 6a1bb577bb1a42418bc6bdb967169edc, State: Running, Role: LEADER
I20260927 22:44:35.596107 17352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:35.596560 17348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:44:35.598809 17354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1bb577bb1a42418bc6bdb967169edc. Latest consensus state: current_term: 2 leader_uuid: "6a1bb577bb1a42418bc6bdb967169edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } } }
I20260927 22:44:35.599207 17354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: This master's current role is: LEADER
I20260927 22:44:35.598902 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6a1bb577bb1a42418bc6bdb967169edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } } }
I20260927 22:44:35.599341 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: This master's current role is: LEADER
I20260927 22:44:35.609776 17359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:44:35.620182 17359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:44:35.622267 17359 catalog_manager.cc:1259] Loaded cluster ID: d5518858836d4aec9efaa3ee7a09fdfe
I20260927 22:44:35.622526 17359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:44:35.628552 17359 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:44:35.630970 17359 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Loaded TSK: 0
I20260927 22:44:35.683624 17359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Generated new TSK 1
I20260927 22:44:35.684069 17359 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:44:35.929118 17350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:35.929436 17350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:35.929584 17350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:35.937381 17350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:35.937511 17350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:35.937690 17350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:44:35.948141 17350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:44225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=44883
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:35.949576 17350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:35.950959 17350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:35.966944 17375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:35.966854 17376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:35.966902 17378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:35.968465 17350 server_base.cc:1034] running on GCE node
I20260927 22:44:35.969283 17350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:35.970496 17350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:35.971766 17350 hybrid_clock.cc:648] HybridClock initialized: now 1790549075971686 us; error 62 us; skew 500 ppm
I20260927 22:44:35.972167 17350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:35 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:35.976401 17350 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:44:35.978206 17350 webserver.cc:460] Webserver started at http://127.1.53.65:44883/ using document root <none> and password file <none>
I20260927 22:44:35.979055 17350 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:35.979229 17350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:35.985229 17350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:44:35.988487 17385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:35.990195 17350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:44:35.990396 17350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "6465cd858a7d439e978b937acf518396"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "2410205ac95c3b41239a2214828b59e5"
server_key_iv: "89f11d8196395033203e941b2a88372e"
server_key_version: "encryptionkey@0"
I20260927 22:44:35.991477 17350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:36.005203 17350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:36.010321 17350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:36.010667 17350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:36.011752 17350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:36.013291 17350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:36.013428 17350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.013561 17350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:36.013648 17350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.049525 17350 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:44225
I20260927 22:44:36.049587 17497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:44225 every 8 connection(s)
I20260927 22:44:36.051146 17350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:44:36.057135 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17350
I20260927 22:44:36.057920 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:33679
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:46099
--webserver_port=40983
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:44:36 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:36.081620 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:36.054751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:50551)
0927 22:44:36.055547 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:36.055587 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:36.058096 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:36.060210 (+ 2114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:36.060223 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:36.061138 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:44:36.061968 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:36.061998 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.063837 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.063846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:36.066162 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:36.066177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.067959 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.067966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:36.068189 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:36.069041 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:44:36.069066 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:44:36.071725 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:44:36.076974 (+ 5249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:36.076982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:36.076998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:36.077513 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:36.078114 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:36.078117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:36.078119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:36.078337 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:36.079428 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:44:36.079434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:44:36.079773 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 22:44:36.080025 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:44:36.080556 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":304,"threads_started":1}
I20260927 22:44:36.082293 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:36.054352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:50551 (local address 127.1.53.126:46099)
0927 22:44:36.054975 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:36.054990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:36.055759 (+ 769us) server_negotiation.cc:407] Connection header received
0927 22:44:36.058976 (+ 3217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:36.059001 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:36.059600 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:36.059986 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:36.062269 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.063622 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.066355 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.067627 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.067859 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:36.072154 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:36.072168 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:36.072195 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:36.072294 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:36.076752 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:36.077708 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:36.077722 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:36.077733 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:36.077989 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:36.078479 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:36.078488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:36.078494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:36.079258 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:36.079515 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:36.080399 (+ 884us) server_negotiation.cc:299] Negotiation successful
0927 22:44:36.080918 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":266,"threads_started":1}
I20260927 22:44:36.085436 17498 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46099
I20260927 22:44:36.085832 17498 heartbeater.cc:464] Registering TS with master...
I20260927 22:44:36.086900 17498 heartbeater.cc:511] Master 127.1.53.126:46099 requested a full tablet report, sending...
I20260927 22:44:36.089571 17313 ts_manager.cc:194] Registered new tserver with Master: 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225)
I20260927 22:44:36.091076 17313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:50551
W20260927 22:44:36.412793 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:36.413106 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:36.413259 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:36.420856 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:36.420992 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:36.421190 17502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:44:36.431720 17502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:33679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=40983
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:36.433174 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:36.434569 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:36.449364 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:36.449479 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:36.449431 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:36.454162 17502 server_base.cc:1034] running on GCE node
I20260927 22:44:36.454934 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:36.456146 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:36.457453 17502 hybrid_clock.cc:648] HybridClock initialized: now 1790549076457383 us; error 63 us; skew 500 ppm
I20260927 22:44:36.457816 17502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:36 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:36.461654 17502 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:44:36.463495 17502 webserver.cc:460] Webserver started at http://127.1.53.66:40983/ using document root <none> and password file <none>
I20260927 22:44:36.464437 17502 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:36.464630 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:36.471041 17502 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:44:36.474382 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.476030 17502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:44:36.476243 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "026402c65d41406ea13c1f24394fbd0e"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "01c26576efe4a250b697474f72369897"
server_key_iv: "c8bd54001a14647646784f6980e06239"
server_key_version: "encryptionkey@0"
I20260927 22:44:36.477502 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:36.495043 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:36.500049 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:36.500413 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:36.501492 17502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:36.502943 17502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:36.503065 17502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.503201 17502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:36.503301 17502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.539618 17502 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:33679
I20260927 22:44:36.539652 17629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:33679 every 8 connection(s)
I20260927 22:44:36.541402 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:44:36.548463 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17502
I20260927 22:44:36.549197 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:44139
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:46099
--webserver_port=46837
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:44:36 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:36.564574 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:36.544416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:50427 (local address 127.1.53.126:46099)
0927 22:44:36.544602 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:36.544607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:36.545968 (+ 1361us) server_negotiation.cc:407] Connection header received
0927 22:44:36.547885 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:36.547891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:36.548027 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:36.548158 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:36.549966 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.550686 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.552373 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.552900 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.553032 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:36.556895 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:36.556902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:36.556905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:36.556939 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:36.560682 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:36.561555 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:36.561564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:36.561571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:36.561736 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:36.562386 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:36.562394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:36.562400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:36.562833 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:36.562991 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:36.563814 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 22:44:36.564100 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:44:36.565241 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:36.544988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:50427)
0927 22:44:36.545780 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:36.545819 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:36.547557 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:36.548427 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:36.548435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:36.549104 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:44:36.549791 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:36.549809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.550864 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.550868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:36.552155 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:36.552167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:36.553273 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:36.553278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:36.553482 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:36.554257 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:44:36.554281 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:44:36.556612 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:44:36.560865 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:36.560873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:36.560892 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:36.561392 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:36.561915 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:36.561920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:36.561924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:36.562220 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:36.562995 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:44:36.563003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:44:36.563411 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 22:44:36.563665 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 22:44:36.564099 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20260927 22:44:36.566808 17630 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46099
I20260927 22:44:36.567188 17630 heartbeater.cc:464] Registering TS with master...
I20260927 22:44:36.568161 17630 heartbeater.cc:511] Master 127.1.53.126:46099 requested a full tablet report, sending...
I20260927 22:44:36.569916 17313 ts_manager.cc:194] Registered new tserver with Master: 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
I20260927 22:44:36.570770 17313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:50427
W20260927 22:44:36.892566 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:36.892877 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:36.892977 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:36.900486 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:36.900609 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:36.900725 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:44:36.910881 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:44139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=46837
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:36.912184 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:36.913518 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:36.927590 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:36.927781 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:36.927775 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:36.928601 17633 server_base.cc:1034] running on GCE node
I20260927 22:44:36.929256 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:36.930393 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:36.931607 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790549076931533 us; error 69 us; skew 500 ppm
I20260927 22:44:36.931948 17633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:36 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:36.935357 17633 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:44:36.937184 17633 webserver.cc:460] Webserver started at http://127.1.53.67:46837/ using document root <none> and password file <none>
I20260927 22:44:36.937909 17633 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:36.938081 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:36.943660 17633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:44:36.946635 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.948196 17633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:44:36.948410 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "d1fa1097793b47078eb58e42f4345906"
format_stamp: "Formatted at 2026-09-27 22:44:14 on dist-test-slave-pp80"
server_key: "f40e6c5a830d139f69fd716e3627bd09"
server_key_iv: "dcc864295818c6006e106fc84800d95e"
server_key_version: "encryptionkey@0"
I20260927 22:44:36.949504 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:36.965751 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:36.970593 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:36.970906 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:36.971979 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:36.973493 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:36.973620 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:36.973771 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:36.973889 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:37.008747 17633 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:44139
I20260927 22:44:37.008841 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:44139 every 8 connection(s)
I20260927 22:44:37.010399 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:44:37.020061 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17633
Sep 27 22:44:37 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549076, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:44:37.032071 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.013341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:45413 (local address 127.1.53.126:46099)
0927 22:44:37.013528 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.013533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.014793 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 22:44:37.016728 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.016734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.016836 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.016956 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:37.018594 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.019339 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.021362 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.022006 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.022085 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.025794 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:37.025800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:37.025803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:37.025836 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:37.028775 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.029512 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.029522 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.029524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.029658 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.030123 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.030127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.030130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.030502 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:37.030685 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.031392 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.031658 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:44:37.033095 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.013874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:45413)
0927 22:44:37.014620 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.014664 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:37.016436 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:37.017261 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:37.017269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:37.017835 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:44:37.018418 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.018435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.019502 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.019507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.021153 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.021170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.022379 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.022384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.022587 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.023309 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:44:37.023340 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:44:37.025580 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:44:37.028915 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:37.028925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:37.028944 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:37.029377 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:37.029774 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:44:37.029779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:44:37.029783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:44:37.029993 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:44:37.030658 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:44:37.030667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:44:37.031039 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 22:44:37.031300 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 22:44:37.031872 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":294,"threads_started":1}
I20260927 22:44:37.034667 17761 heartbeater.cc:347] Connected to a master server at 127.1.53.126:46099
I20260927 22:44:37.035009 17761 heartbeater.cc:464] Registering TS with master...
I20260927 22:44:37.035830 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.021086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.1.53.126:46099)
0927 22:44:37.021681 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.021688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.021931 (+ 243us) server_negotiation.cc:407] Connection header received
0927 22:44:37.022136 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.022143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.022214 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.022324 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:37.023855 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.024667 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.026153 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.026616 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.026754 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.029522 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:37.029527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:37.029530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:37.029562 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:37.032630 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.033426 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.033431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.033436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.033550 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.033963 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.033968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.033972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.034503 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:37.034840 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.035201 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.035432 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":279,"threads_started":1}
I20260927 22:44:37.035851 17761 heartbeater.cc:511] Master 127.1.53.126:46099 requested a full tablet report, sending...
I20260927 22:44:37.037705 17313 ts_manager.cc:194] Registered new tserver with Master: d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139)
I20260927 22:44:37.038563 17313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:45413
I20260927 22:44:37.039034 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:44:37.045949 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.039986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.1.53.126:46099)
0927 22:44:37.040166 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.040173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.040679 (+ 506us) server_negotiation.cc:407] Connection header received
0927 22:44:37.040945 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.040951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.041089 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.041237 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:44:37.041667 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.042355 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.043869 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.044442 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.044550 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.045021 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:44:37.045243 (+ 222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:44:37.045650 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43942: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 22:44:37.046046 17764 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43942: token expired
W20260927 22:44:37.046634 17235 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.53.126:46099)
I20260927 22:44:37.060720 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.047148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.1.53.126:46099)
0927 22:44:37.047338 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.047346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.047366 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:44:37.047577 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.047581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.047641 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.047740 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:37.049263 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.049968 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.051377 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.051791 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.051924 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.054932 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:37.054938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:37.054941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:37.054976 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:37.057800 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.058599 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.058605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.058609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.058755 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.059157 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.059164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.059172 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.059503 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:37.059759 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.060075 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.060372 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:44:37.064064 17313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:44:37.071482 17313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:44:37.095170 17498 heartbeater.cc:503] Master 127.1.53.126:46099 was elected leader, sending a full tablet report...
I20260927 22:44:37.097476 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.084267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:44139 (local address 127.0.0.1:38344)
0927 22:44:37.085271 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.085303 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:37.085500 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:37.086654 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:37.086665 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:37.086707 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:37.087242 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.087257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.089973 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.089982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.093404 (+ 3422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.093441 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.096295 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.096307 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.096636 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.096714 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:44:37.096869 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:44:37.097091 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":593,"threads_started":1}
I20260927 22:44:37.097919 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.084354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.1.53.67:44139)
0927 22:44:37.085125 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.085134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.085413 (+ 279us) server_negotiation.cc:407] Connection header received
0927 22:44:37.085781 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.085794 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.086103 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.086313 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:37.087522 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.089697 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.093695 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.095548 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.095704 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.096040 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.097374 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.097576 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":422,"threads_started":1}
I20260927 22:44:37.102322 17696 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6b2916f264ebcbf70a06605a164e2 (DEFAULT_TABLE table=test-table [id=5373970e4fd34983a301225abb70903e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:44:37.104022 17696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6b2916f264ebcbf70a06605a164e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:44:37.107810 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.085619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52214)
0927 22:44:37.086223 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.086246 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:37.086386 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:37.100313 (+ 13927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:37.100318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:37.100349 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:37.100738 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.100751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.102859 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.102864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.104999 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.105021 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.106937 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.106946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.107137 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.107175 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:44:37.107281 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:44:37.107431 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":289,"threads_started":1}
I20260927 22:44:37.108996 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.085960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.1.53.65:44225)
0927 22:44:37.099592 (+ 13632us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.099601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.099640 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:44:37.099742 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.099750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.100049 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.100941 (+ 892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:37.101161 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.102694 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.105180 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.106692 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.106849 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.107105 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.108473 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.108673 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13448,"thread_start_us":13315,"threads_started":1}
I20260927 22:44:37.109674 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.086268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33679 (local address 127.0.0.1:37042)
0927 22:44:37.096885 (+ 10617us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.096904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:37.097012 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:37.102873 (+ 5861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:37.102878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:37.102916 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:37.103194 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.103202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.104669 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.104677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.106692 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.106708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.108894 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.108901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.109078 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.109116 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:44:37.109204 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:44:37.109341 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10460,"thread_start_us":10823,"threads_started":1}
I20260927 22:44:37.109889 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.086752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.1.53.66:33679)
0927 22:44:37.102084 (+ 15332us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.102093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.102121 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:44:37.102219 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.102226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.102487 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.102671 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:37.103378 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.104535 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.106913 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.108671 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.108823 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.109032 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.109359 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.109537 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15153,"thread_start_us":15830,"threads_started":1}
I20260927 22:44:37.113243 17565 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6b2916f264ebcbf70a06605a164e2 (DEFAULT_TABLE table=test-table [id=5373970e4fd34983a301225abb70903e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:44:37.113380 17433 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6b2916f264ebcbf70a06605a164e2 (DEFAULT_TABLE table=test-table [id=5373970e4fd34983a301225abb70903e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:44:37.114653 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6b2916f264ebcbf70a06605a164e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:44:37.114807 17433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6b2916f264ebcbf70a06605a164e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:44:37.120677 17777 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Bootstrap starting.
I20260927 22:44:37.125322 17777 tablet_bootstrap.cc:654] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:37.126729 17777 log.cc:824] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:37.129575 17779 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Bootstrap starting.
I20260927 22:44:37.131024 17777 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: No bootstrap required, opened a new log
I20260927 22:44:37.131395 17777 ts_tablet_manager.cc:1397] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 22:44:37.133024 17780 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Bootstrap starting.
I20260927 22:44:37.133474 17779 tablet_bootstrap.cc:654] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:37.136286 17779 log.cc:824] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:37.137089 17780 tablet_bootstrap.cc:654] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Neither blocks nor log segments found. Creating new log.
I20260927 22:44:37.138340 17780 log.cc:824] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:37.139921 17779 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: No bootstrap required, opened a new log
I20260927 22:44:37.140228 17779 ts_tablet_manager.cc:1397] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260927 22:44:37.141139 17780 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: No bootstrap required, opened a new log
I20260927 22:44:37.141404 17780 ts_tablet_manager.cc:1397] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:44:37.141916 17777 raft_consensus.cc:348] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.142391 17777 raft_consensus.cc:374] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:37.142550 17777 raft_consensus.cc:729] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fa1097793b47078eb58e42f4345906, State: Initialized, Role: FOLLOWER
I20260927 22:44:37.143249 17777 consensus_queue.cc:260] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.144577 17761 heartbeater.cc:503] Master 127.1.53.126:46099 was elected leader, sending a full tablet report...
I20260927 22:44:37.145241 17777 ts_tablet_manager.cc:1428] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 22:44:37.150642 17779 raft_consensus.cc:348] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.150506 17780 raft_consensus.cc:348] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.151041 17779 raft_consensus.cc:374] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:37.151057 17780 raft_consensus.cc:374] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:44:37.151161 17780 raft_consensus.cc:729] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026402c65d41406ea13c1f24394fbd0e, State: Initialized, Role: FOLLOWER
I20260927 22:44:37.151163 17779 raft_consensus.cc:729] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6465cd858a7d439e978b937acf518396, State: Initialized, Role: FOLLOWER
I20260927 22:44:37.151856 17780 consensus_queue.cc:260] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.151840 17779 consensus_queue.cc:260] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.153119 17630 heartbeater.cc:503] Master 127.1.53.126:46099 was elected leader, sending a full tablet report...
I20260927 22:44:37.153842 17779 ts_tablet_manager.cc:1428] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:44:37.153851 17780 ts_tablet_manager.cc:1428] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:44:37.229120 17783 raft_consensus.cc:482] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:44:37.229416 17783 raft_consensus.cc:504] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.231298 17783 leader_election.cc:288] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225), 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
I20260927 22:44:37.237443 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.231737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.1.53.67:51143)
0927 22:44:37.231947 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.231978 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:37.232108 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:37.232622 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:37.232643 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:37.232676 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:37.232870 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.232878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.233944 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.233951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.235258 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.235271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.236631 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.236637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.236820 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.236845 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:44:37.236996 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 22:44:37.237142 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 22:44:37.237426 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.231895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:51143 (local address 127.1.53.65:44225)
0927 22:44:37.232130 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.232138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.232165 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:44:37.232279 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.232288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.232378 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.232486 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:37.233049 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.233805 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.235453 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.236439 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.236600 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.236706 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.237004 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.237148 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:44:37.238408 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.232470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:43005 (local address 127.1.53.66:33679)
0927 22:44:37.232684 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.232690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.233110 (+ 420us) server_negotiation.cc:407] Connection header received
0927 22:44:37.233303 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.233315 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.233376 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.233479 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:44:37.233996 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.234704 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.236530 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.237432 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.237602 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.237728 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.238024 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.238156 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:44:37.238453 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.232147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33679 (local address 127.1.53.67:43005)
0927 22:44:37.232937 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.232952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:44:37.233158 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:44:37.233489 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:44:37.233494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:44:37.233516 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:44:37.233841 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.233849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.234840 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.234846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.236356 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:44:37.236368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.237644 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.237650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:44:37.237846 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.237870 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:44:37.237987 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:44:37.238118 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":544,"threads_started":1}
I20260927 22:44:37.239117 17453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6b2916f264ebcbf70a06605a164e2" candidate_uuid: "d1fa1097793b47078eb58e42f4345906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6465cd858a7d439e978b937acf518396" is_pre_election: true
I20260927 22:44:37.239778 17453 raft_consensus.cc:2454] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1fa1097793b47078eb58e42f4345906 in term 0.
I20260927 22:44:37.239704 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6b2916f264ebcbf70a06605a164e2" candidate_uuid: "d1fa1097793b47078eb58e42f4345906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026402c65d41406ea13c1f24394fbd0e" is_pre_election: true
I20260927 22:44:37.240213 17585 raft_consensus.cc:2454] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1fa1097793b47078eb58e42f4345906 in term 0.
I20260927 22:44:37.240563 17652 leader_election.cc:302] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6465cd858a7d439e978b937acf518396, d1fa1097793b47078eb58e42f4345906; no voters:
I20260927 22:44:37.241055 17783 raft_consensus.cc:2781] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:44:37.241212 17783 raft_consensus.cc:482] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:44:37.241328 17783 raft_consensus.cc:3037] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:37.243592 17783 raft_consensus.cc:504] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.244305 17783 leader_election.cc:288] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [CANDIDATE]: Term 1 election: Requested vote from peers 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225), 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
I20260927 22:44:37.244827 17453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6b2916f264ebcbf70a06605a164e2" candidate_uuid: "d1fa1097793b47078eb58e42f4345906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6465cd858a7d439e978b937acf518396"
I20260927 22:44:37.244871 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6b2916f264ebcbf70a06605a164e2" candidate_uuid: "d1fa1097793b47078eb58e42f4345906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026402c65d41406ea13c1f24394fbd0e"
I20260927 22:44:37.245075 17453 raft_consensus.cc:3037] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:37.245082 17585 raft_consensus.cc:3037] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:44:37.247537 17585 raft_consensus.cc:2454] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1fa1097793b47078eb58e42f4345906 in term 1.
I20260927 22:44:37.247579 17453 raft_consensus.cc:2454] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1fa1097793b47078eb58e42f4345906 in term 1.
I20260927 22:44:37.247991 17652 leader_election.cc:302] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906; no voters:
I20260927 22:44:37.248312 17783 raft_consensus.cc:2781] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:44:37.248832 17783 raft_consensus.cc:686] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 1 LEADER]: Becoming Leader. State: Replica: d1fa1097793b47078eb58e42f4345906, State: Running, Role: LEADER
I20260927 22:44:37.249411 17783 consensus_queue.cc:237] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:37.254434 17312 catalog_manager.cc:5495] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1fa1097793b47078eb58e42f4345906 (127.1.53.67). New cstate: current_term: 1 leader_uuid: "d1fa1097793b47078eb58e42f4345906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } health_report { overall_health: UNKNOWN } } }
W20260927 22:44:37.262091 17762 tablet.cc:2369] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:44:37.290534 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.285119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.1.53.67:44139)
0927 22:44:37.285322 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.285327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.285345 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:44:37.285508 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.285513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.285594 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.285725 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:44:37.286141 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.286849 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.288386 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.288785 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.288884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.289363 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:44:37.289885 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:44:37.290008 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.290149 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.290221 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 22:44:37.293560 17631 tablet.cc:2369] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:44:37.299430 17453 raft_consensus.cc:1261] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Refusing update from remote peer d1fa1097793b47078eb58e42f4345906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:44:37.299439 17585 raft_consensus.cc:1261] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Refusing update from remote peer d1fa1097793b47078eb58e42f4345906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:44:37.300587 17787 consensus_queue.cc:1035] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:44:37.301041 17783 consensus_queue.cc:1035] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:44:37.303716 17499 tablet.cc:2369] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:44:37.334528 17313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:44:37.334955 17313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:44:37.339046 17313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Sending DeleteTablet for 3 replicas of tablet 31f6b2916f264ebcbf70a06605a164e2
I20260927 22:44:37.340370 17696 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6b2916f264ebcbf70a06605a164e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:44:37 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:38344
I20260927 22:44:37.340627 17565 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6b2916f264ebcbf70a06605a164e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:44:37 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:37042
I20260927 22:44:37.340746 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 17350
I20260927 22:44:37.340878 17433 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6b2916f264ebcbf70a06605a164e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:44:37 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:52214
I20260927 22:44:37.353096 17799 tablet_replica.cc:330] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: stopping tablet replica
I20260927 22:44:37.353404 17800 tablet_replica.cc:330] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: stopping tablet replica
I20260927 22:44:37.353875 17799 raft_consensus.cc:2229] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:44:37.353995 17801 tablet_replica.cc:330] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: stopping tablet replica
I20260927 22:44:37.354540 17801 raft_consensus.cc:2229] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:44:37.354807 17799 raft_consensus.cc:2258] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:44:37.355087 17800 raft_consensus.cc:2229] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:44:37.355715 17801 raft_consensus.cc:2258] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:44:37.356283 17800 raft_consensus.cc:2258] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:44:37.357585 17799 ts_tablet_manager.cc:1905] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:44:37.359385 17801 ts_tablet_manager.cc:1905] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:44:37.361419 17800 ts_tablet_manager.cc:1905] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:44:37.362068 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.341714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.1.53.65:44225)
0927 22:44:37.342523 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.342531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.342554 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:44:37.342626 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.342634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.342771 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.342916 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:37.345417 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.346845 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.349057 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.349772 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.349879 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.354114 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:37.354137 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:37.354160 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:37.354209 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:37.358360 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.359377 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.359387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.359391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.359560 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.360063 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.360068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.360071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.360558 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:37.360884 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.361451 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.361699 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635}
I20260927 22:44:37.362993 17493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:37.367689 17799 ts_tablet_manager.cc:1918] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:44:37.367957 17799 log.cc:1196] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/31f6b2916f264ebcbf70a06605a164e2
I20260927 22:44:37.368659 17799 ts_tablet_manager.cc:1939] T 31f6b2916f264ebcbf70a06605a164e2 P d1fa1097793b47078eb58e42f4345906: Deleting consensus metadata
I20260927 22:44:37.370820 17300 catalog_manager.cc:4841] TS d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139): tablet 31f6b2916f264ebcbf70a06605a164e2 (table test-table [id=5373970e4fd34983a301225abb70903e]) successfully deleted
I20260927 22:44:37.374007 17800 ts_tablet_manager.cc:1918] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:44:37.374210 17800 log.cc:1196] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/31f6b2916f264ebcbf70a06605a164e2
I20260927 22:44:37.374672 17800 ts_tablet_manager.cc:1939] T 31f6b2916f264ebcbf70a06605a164e2 P 026402c65d41406ea13c1f24394fbd0e: Deleting consensus metadata
I20260927 22:44:37.376354 17300 catalog_manager.cc:4841] TS 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679): tablet 31f6b2916f264ebcbf70a06605a164e2 (table test-table [id=5373970e4fd34983a301225abb70903e]) successfully deleted
I20260927 22:44:37.482787 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 17350
W20260927 22:44:37.501381 17300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:44225 (error 108)
I20260927 22:44:37.501551 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 17502
I20260927 22:44:37.503610 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.503095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52236)
0927 22:44:37.503358 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.503472 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260927 22:44:37.503897 17300 catalog_manager.cc:4584] TS 6465cd858a7d439e978b937acf518396 (127.1.53.65:44225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31f6b2916f264ebcbf70a06605a164e2: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
I20260927 22:44:37.518023 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.502417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.1.53.66:33679)
0927 22:44:37.502606 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.502615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.502736 (+ 121us) server_negotiation.cc:407] Connection header received
0927 22:44:37.502886 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.502892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.502999 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.503111 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:37.504795 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.505567 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.507559 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.507989 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.508094 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.511447 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:37.511461 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:37.511476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:37.511511 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:37.514801 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.515637 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.515644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.515647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.515770 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.516216 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.516220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.516222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.516863 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:37.517112 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.517499 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.517673 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:44:37.518756 17625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:37.557624 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.557018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52238)
0927 22:44:37.557315 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.557423 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260927 22:44:37.558902 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.558465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52242)
0927 22:44:37.558640 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.558731 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:44:37.619885 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.619285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52256)
0927 22:44:37.619551 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.619673 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 22:44:37.621140 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.620692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52260)
0927 22:44:37.620876 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.620984 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:44:37.622877 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 17502
I20260927 22:44:37.642419 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 17633
I20260927 22:44:37.659893 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.643428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.1.53.67:44139)
0927 22:44:37.643648 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:44:37.643654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:44:37.643678 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:44:37.643752 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:44:37.643758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:44:37.643860 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:44:37.643995 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:44:37.645520 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.646774 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.648485 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:44:37.649171 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:44:37.649340 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:44:37.652200 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:44:37.652218 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:44:37.652264 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:44:37.652335 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:44:37.656586 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.657392 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.657401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.657404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.657606 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:44:37.658050 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:44:37.658055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:44:37.658059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:44:37.658554 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:44:37.658940 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 22:44:37.659218 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:44:37.659478 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:44:37.660871 17756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:37.702756 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.702180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52266)
0927 22:44:37.702446 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.702551 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:44:37.704113 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:37.703610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:44225 (local address 127.0.0.1:52270)
0927 22:44:37.703820 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:37.703917 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 22:44:37.789829 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 17633
I20260927 22:44:37.812111 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 17280
I20260927 22:44:37.813439 17343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:44:37.933539 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 17280
I20260927 22:44:57.956105 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46099
--webserver_interface=127.1.53.126
--webserver_port=40781
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:58.320787 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:58.321184 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:58.321360 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:58.328965 17807 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:44:58.329147 17807 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:44:58.329248 17807 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:44:58.329347 17807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:44:58.329430 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:58.329535 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:58.329629 17807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:44:58.329711 17807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:44:58.340372 17807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:58.341804 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:58.343212 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:58.356372 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:58.356204 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:58.356225 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:58.356998 17807 server_base.cc:1034] running on GCE node
I20260927 22:44:58.357909 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:58.359493 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:58.360764 17807 hybrid_clock.cc:648] HybridClock initialized: now 1790549098360704 us; error 46 us; skew 500 ppm
I20260927 22:44:58.361162 17807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:58 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549098, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:58.374014 17807 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:44:58.375856 17807 webserver.cc:460] Webserver started at http://127.1.53.126:40781/ using document root <none> and password file <none>
I20260927 22:44:58.376783 17807 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:58.376936 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:58.382969 17807 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:44:58.386305 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:58.387743 17807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:44:58.387946 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "6a1bb577bb1a42418bc6bdb967169edc"
format_stamp: "Formatted at 2026-09-27 22:44:12 on dist-test-slave-pp80"
server_key: "f6481053abd69b34f0f7cd533b7928a6"
server_key_iv: "bfb031a6d8ba4884fb2c515a7579b256"
server_key_version: "encryptionkey@0"
I20260927 22:44:58.389062 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:58.417865 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:58.422638 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:58.422950 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:58.443111 17807 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:46099
I20260927 22:44:58.443197 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:46099 every 8 connection(s)
I20260927 22:44:58.445218 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:44:58.452194 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17807
I20260927 22:44:58.453141 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:44225
--local_ip_for_outbound_sockets=127.1.53.65
--tserver_master_addrs=127.1.53.126:46099
--webserver_port=44883
--webserver_interface=127.1.53.65
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:44:58.453613 17874 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:44:58.457409 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap starting.
I20260927 22:44:58.480559 17874 log.cc:824] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:58.497241 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:44:58.497843 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Bootstrap complete.
I20260927 22:44:58.507280 17874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:58.508293 17874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bb577bb1a42418bc6bdb967169edc, State: Initialized, Role: FOLLOWER
I20260927 22:44:58.508984 17874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:58.509239 17874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:44:58.509375 17874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:44:58.509609 17874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:44:58.513598 17874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:58.514134 17874 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1bb577bb1a42418bc6bdb967169edc; no voters:
I20260927 22:44:58.514833 17874 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:44:58.514994 17878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:44:58.516052 17878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [term 3 LEADER]: Becoming Leader. State: Replica: 6a1bb577bb1a42418bc6bdb967169edc, State: Running, Role: LEADER
I20260927 22:44:58.516710 17878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } }
I20260927 22:44:58.517120 17874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:44:58.519412 17880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1bb577bb1a42418bc6bdb967169edc. Latest consensus state: current_term: 3 leader_uuid: "6a1bb577bb1a42418bc6bdb967169edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } } }
I20260927 22:44:58.519922 17880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: This master's current role is: LEADER
I20260927 22:44:58.519480 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6a1bb577bb1a42418bc6bdb967169edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1bb577bb1a42418bc6bdb967169edc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46099 } } }
I20260927 22:44:58.520159 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc [sys.catalog]: This master's current role is: LEADER
I20260927 22:44:58.533998 17885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:44:58.543740 17885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:44:58.545549 17885 catalog_manager.cc:1259] Loaded cluster ID: d5518858836d4aec9efaa3ee7a09fdfe
I20260927 22:44:58.545753 17885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:44:58.550021 17885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:44:58.551813 17885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Loaded TSK: 0
I20260927 22:44:58.552065 17885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a1bb577bb1a42418bc6bdb967169edc: Loaded TSK: 1
F20260927 22:44:58.553205 17885 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc98bd3d9ad google::LogMessage::Fail() at ??:0
@ 0x7fc98bd4088a google::LogMessage::SendToLog() at ??:0
@ 0x7fc98bd3d4de google::LogMessage::Flush() at ??:0
@ 0x7fc98bd3ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc98faa17d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc98fa9073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc9a17721c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc9a1771783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
I20260927 22:44:58.588105 17897 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fc9a17fd2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fc9a17fd0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc9a18475e9 std::function<>::operator()() at ??:0
@ 0x7fc9a1775cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc9a1773c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc9a17fb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc9a17fb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc98e989345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc98e98c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc98e98c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc98e967eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc991c416db start_thread at ??:0
@ 0x7fc98abd761f clone at ??:0
W20260927 22:44:58.866262 17876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:58.866677 17876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:58.866871 17876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:58.875326 17876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:58.875519 17876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:58.875710 17876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:44:58.886468 17876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:44225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=44883
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:58.887924 17876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:58.889376 17876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:58.903403 17921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:58.903596 17924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:58.903595 17922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:58.904610 17876 server_base.cc:1034] running on GCE node
I20260927 22:44:58.905349 17876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:58.906558 17876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:58.907809 17876 hybrid_clock.cc:648] HybridClock initialized: now 1790549098907752 us; error 55 us; skew 500 ppm
I20260927 22:44:58.908190 17876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:58 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549098, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:58.911767 17876 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:44:58.913550 17876 webserver.cc:460] Webserver started at http://127.1.53.65:44883/ using document root <none> and password file <none>
I20260927 22:44:58.914297 17876 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:58.914438 17876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:58.920714 17876 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:44:58.924582 17931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:58.926257 17876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:44:58.926476 17876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "6465cd858a7d439e978b937acf518396"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "2410205ac95c3b41239a2214828b59e5"
server_key_iv: "89f11d8196395033203e941b2a88372e"
server_key_version: "encryptionkey@0"
I20260927 22:44:58.927543 17876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:58.947795 17876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:58.952590 17876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:58.952893 17876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:58.953961 17876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:58.955888 17938 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 22:44:58.962548 17876 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 22:44:58.962706 17876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 22:44:58.962857 17876 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 22:44:58.966235 17876 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 22:44:58.966358 17876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260927 22:44:58.966741 17938 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Bootstrap starting.
I20260927 22:44:58.976517 17938 log.cc:824] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Log is configured to *not* fsync() on all Append() calls
I20260927 22:44:58.997211 17938 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:44:58.998209 17938 tablet_bootstrap.cc:492] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Bootstrap complete.
I20260927 22:44:58.998982 17938 ts_tablet_manager.cc:1397] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.000s
I20260927 22:44:59.006237 17876 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:44225
I20260927 22:44:59.006276 18045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:44225 every 8 connection(s)
I20260927 22:44:59.008090 17876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:44:59.010316 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 17876
I20260927 22:44:59.010985 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:33679
--local_ip_for_outbound_sockets=127.1.53.66
--tserver_master_addrs=127.1.53.126:46099
--webserver_port=40983
--webserver_interface=127.1.53.66
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:44:59.011428 17938 raft_consensus.cc:348] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:59.012496 17938 raft_consensus.cc:729] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6465cd858a7d439e978b937acf518396, State: Initialized, Role: FOLLOWER
I20260927 22:44:59.013285 17938 consensus_queue.cc:260] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:44:59.015318 17938 ts_tablet_manager.cc:1428] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 22:44:59.017120 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.014432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:49975)
0927 22:44:59.015554 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.015869 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":443,"threads_started":1}
W20260927 22:44:59.019680 18046 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:46099 (0 consecutive failures): Network error: Failed to ping master at 127.1.53.126:46099: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
I20260927 22:44:59.020795 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.020312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:48491)
0927 22:44:59.020524 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.020628 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:44:59.021826 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.021361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:36563)
0927 22:44:59.021555 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.021657 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:44:59.022101 18046 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:44:59.356932 18050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:59.357241 18050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:59.357345 18050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:59.364883 18050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:59.365023 18050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:59.365206 18050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:44:59.375432 18050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:33679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=40983
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:59.376873 18050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:59.378376 18050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:59.392474 18055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:59.392517 18056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:59.392558 18058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:59.393435 18050 server_base.cc:1034] running on GCE node
I20260927 22:44:59.394248 18050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:59.395295 18050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:59.396551 18050 hybrid_clock.cc:648] HybridClock initialized: now 1790549099396480 us; error 74 us; skew 500 ppm
I20260927 22:44:59.396901 18050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:59 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549099, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:59.400301 18050 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:44:59.401976 18050 webserver.cc:460] Webserver started at http://127.1.53.66:40983/ using document root <none> and password file <none>
I20260927 22:44:59.402750 18050 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:59.402908 18050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:59.409139 18050 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:44:59.412561 18065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:59.414405 18050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:44:59.414626 18050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "026402c65d41406ea13c1f24394fbd0e"
format_stamp: "Formatted at 2026-09-27 22:44:13 on dist-test-slave-pp80"
server_key: "01c26576efe4a250b697474f72369897"
server_key_iv: "c8bd54001a14647646784f6980e06239"
server_key_version: "encryptionkey@0"
I20260927 22:44:59.415736 18050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:59.438447 18050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:59.443190 18050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:59.443500 18050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:59.444669 18050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:59.446113 18050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:59.446245 18050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:59.446396 18050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:59.446494 18050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:59.481665 18050 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:33679
I20260927 22:44:59.481756 18177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:33679 every 8 connection(s)
I20260927 22:44:59.483168 18050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:44:59.487641 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18050
I20260927 22:44:59.488349 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.486236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:58437)
0927 22:44:59.486967 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.487207 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":286,"threads_started":1}
I20260927 22:44:59.488483 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:44139
--local_ip_for_outbound_sockets=127.1.53.67
--tserver_master_addrs=127.1.53.126:46099
--webserver_port=46837
--webserver_interface=127.1.53.67
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:44:59.489027 18178 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:46099 (0 consecutive failures): Network error: Failed to ping master at 127.1.53.126:46099: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
I20260927 22:44:59.489892 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.489548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:42743)
0927 22:44:59.489707 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.489786 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:44:59.490677 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.490352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:38063)
0927 22:44:59.490492 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.490571 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:44:59.490877 18178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:44:59.836621 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:44:59.836913 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:44:59.837016 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:44:59.844676 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:44:59.844805 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:44:59.844925 18181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:44:59.855230 18181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:44139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=46837
--tserver_master_addrs=127.1.53.126:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:44:59.856577 18181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:44:59.857904 18181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:44:59.872092 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:59.872337 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:44:59.872337 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:44:59.873145 18181 server_base.cc:1034] running on GCE node
I20260927 22:44:59.873853 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:44:59.874960 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:44:59.876206 18181 hybrid_clock.cc:648] HybridClock initialized: now 1790549099876160 us; error 38 us; skew 500 ppm
I20260927 22:44:59.876639 18181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:44:59 dist-test-slave-pp80 krb5kdc[16728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549099, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:44:59.880182 18181 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:44:59.881855 18181 webserver.cc:460] Webserver started at http://127.1.53.67:46837/ using document root <none> and password file <none>
I20260927 22:44:59.882632 18181 fs_manager.cc:359] Metadata directory not provided
I20260927 22:44:59.882792 18181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:44:59.889096 18181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:44:59.892161 18196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:59.893774 18181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:44:59.893999 18181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "d1fa1097793b47078eb58e42f4345906"
format_stamp: "Formatted at 2026-09-27 22:44:14 on dist-test-slave-pp80"
server_key: "f40e6c5a830d139f69fd716e3627bd09"
server_key_iv: "dcc864295818c6006e106fc84800d95e"
server_key_version: "encryptionkey@0"
I20260927 22:44:59.895078 18181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:44:59.915279 18181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:44:59.919816 18181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:44:59.920120 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:44:59.921334 18181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:44:59.922820 18181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:44:59.922945 18181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:59.923090 18181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:44:59.923170 18181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:44:59.957778 18181 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:44139
I20260927 22:44:59.957870 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:44139 every 8 connection(s)
I20260927 22:44:59.959275 18181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:44:59.964143 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18181
I20260927 22:44:59.964329 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.962454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:48697)
0927 22:44:59.963184 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.963386 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":284,"threads_started":1}
W20260927 22:44:59.965181 18309 heartbeater.cc:644] Failed to heartbeat to 127.1.53.126:46099 (0 consecutive failures): Network error: Failed to ping master at 127.1.53.126:46099: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
I20260927 22:44:59.966161 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.965786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:43079)
0927 22:44:59.965940 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.966005 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:44:59.966970 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:44:59.966682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:38199)
0927 22:44:59.966799 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 22:44:59.966868 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:44:59.967197 18309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:45:00.023447 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.022672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:37771)
0927 22:45:00.023142 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:00.023230 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":208,"threads_started":1}
I20260927 22:45:00.450555 18314 raft_consensus.cc:482] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:00.451032 18314 raft_consensus.cc:504] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:45:00.453329 18314 leader_election.cc:288] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139), 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
Sep 27 22:45:00 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549098, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
Sep 27 22:45:00 dist-test-slave-pp80 krb5kdc[16728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549098, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:45:00.479167 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.453725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:44139 (local address 127.1.53.65:35751)
0927 22:45:00.453925 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:00.453972 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:00.454574 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:00.458281 (+ 3707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:00.458289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:00.459120 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:00.459899 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:00.459917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.461927 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.461932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:00.463662 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:00.463678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.464918 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.464930 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:00.465114 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:00.465947 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:00.466005 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:00.469148 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:00.474218 (+ 5070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:00.474230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:00.474250 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:00.475064 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:00.475686 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:00.475691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:00.475695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:00.475974 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:00.477108 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:00.477118 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:00.477656 (+ 538us) client_negotiation.cc:769] Sending connection context
0927 22:45:00.478004 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 22:45:00.478647 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":284,"spinlock_wait_cycles":75520}
I20260927 22:45:00.479235 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.453919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:35751 (local address 127.1.53.67:44139)
0927 22:45:00.455565 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:00.455583 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:00.455632 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:45:00.457368 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:00.457377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:00.457778 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:00.458145 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:00.460124 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.461742 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.463819 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.464664 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.464883 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:00.469450 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:00.469460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:00.469475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:00.469554 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:00.473998 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:00.475247 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:00.475258 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:00.475268 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:00.475540 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:00.476105 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:00.476111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:00.476115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:00.476937 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:00.477153 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:00.478289 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 22:45:00.478754 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":1220,"threads_started":1}
I20260927 22:45:00.479246 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.454703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:45699 (local address 127.1.53.66:33679)
0927 22:45:00.455808 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:00.455818 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:00.455849 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:45:00.457350 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:00.457358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:00.457782 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:00.458124 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:00.460107 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.461360 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.463868 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.464588 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.464806 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:00.469348 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:00.469364 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:00.469390 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:00.469463 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:00.473998 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:00.475247 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:00.475258 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:00.475268 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:00.475532 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:00.476105 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:00.476111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:00.476115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:00.476936 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:00.477218 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:00.478288 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0927 22:45:00.478754 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":851,"threads_started":1}
I20260927 22:45:00.479189 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.454154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:33679 (local address 127.1.53.65:45699)
0927 22:45:00.455170 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:00.455189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:00.455320 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:00.458277 (+ 2957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:00.458288 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:00.459121 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:00.459895 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:00.459916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.461547 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.461552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:00.463662 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:00.463678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:00.464801 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:00.464808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:00.465091 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:00.466004 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:00.466022 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:00.468895 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:00.474218 (+ 5323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:00.474230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:00.474250 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:00.475064 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:00.475686 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:00.475691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:00.475695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:00.475974 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:00.477143 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:00.477147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:00.477656 (+ 509us) client_negotiation.cc:769] Sending connection context
0927 22:45:00.478004 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 22:45:00.478747 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":442,"spinlock_wait_cycles":35840,"thread_start_us":682,"threads_started":1}
W20260927 22:45:00.483609 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
W20260927 22:45:00.485088 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
I20260927 22:45:00.485299 17935 leader_election.cc:302] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6465cd858a7d439e978b937acf518396; no voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906
I20260927 22:45:00.485738 18314 raft_consensus.cc:2726] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:45:00.492429 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.491450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:54971)
0927 22:45:00.491914 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:00.492057 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":224,"threads_started":1}
I20260927 22:45:00.968890 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:00.967778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:53389)
0927 22:45:00.968404 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:00.968608 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
I20260927 22:45:01.025352 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:01.024581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:41501)
0927 22:45:01.025059 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:01.025169 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260927 22:45:01.494654 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:01.493546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:43489)
0927 22:45:01.494265 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:01.494367 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":328,"threads_started":1}
I20260927 22:45:01.970559 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:01.969878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:42883)
0927 22:45:01.970315 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:01.970395 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260927 22:45:02.027061 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:02.026294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:32963)
0927 22:45:02.026784 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:02.026864 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
I20260927 22:45:02.279816 18324 raft_consensus.cc:482] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:02.280004 18324 raft_consensus.cc:504] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:45:02.280777 18324 leader_election.cc:288] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139), 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
W20260927 22:45:02.281929 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
W20260927 22:45:02.282143 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
I20260927 22:45:02.282219 17935 leader_election.cc:302] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6465cd858a7d439e978b937acf518396; no voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906
I20260927 22:45:02.282454 18324 raft_consensus.cc:2726] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:45:02.496601 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:02.495735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:39439)
0927 22:45:02.496320 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:02.496419 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":295,"threads_started":1}
I20260927 22:45:02.972265 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:02.971545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:42049)
0927 22:45:02.971988 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:02.972068 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260927 22:45:03.028965 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:03.028177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:52117)
0927 22:45:03.028704 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:03.028792 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":292,"threads_started":1}
I20260927 22:45:03.498425 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:03.497648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:46859)
0927 22:45:03.498154 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:03.498238 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":242,"threads_started":1}
I20260927 22:45:03.973984 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:03.973204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:53935)
0927 22:45:03.973722 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:03.973803 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":213,"threads_started":1}
I20260927 22:45:04.030884 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:04.030025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:39895)
0927 22:45:04.030592 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:04.030697 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":261,"threads_started":1}
I20260927 22:45:04.077248 18331 raft_consensus.cc:482] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:04.077523 18331 raft_consensus.cc:504] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:45:04.078186 18331 leader_election.cc:288] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139), 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
W20260927 22:45:04.079499 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
W20260927 22:45:04.079753 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
I20260927 22:45:04.079905 17935 leader_election.cc:302] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6465cd858a7d439e978b937acf518396; no voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906
I20260927 22:45:04.080300 18331 raft_consensus.cc:2726] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:45:04.500464 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:04.499414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:54367)
0927 22:45:04.500123 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:04.500210 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":361,"threads_started":1}
I20260927 22:45:04.975688 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:04.974992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:50763)
0927 22:45:04.975428 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:04.975518 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260927 22:45:05.032810 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:05.031911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:46751)
0927 22:45:05.032486 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:05.032604 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":257,"threads_started":1}
I20260927 22:45:05.502215 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:05.501426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:39239)
0927 22:45:05.501921 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:05.502009 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260927 22:45:05.977559 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:05.976821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:56821)
0927 22:45:05.977277 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:05.977363 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260927 22:45:06.034668 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:06.033864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:32941)
0927 22:45:06.034329 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:06.034425 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260927 22:45:06.504333 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:06.503343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:42651)
0927 22:45:06.503965 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:06.504067 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":307,"threads_started":1}
I20260927 22:45:06.979619 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:06.978689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:40721)
0927 22:45:06.979290 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:06.979390 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":263,"threads_started":1}
I20260927 22:45:07.036770 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:07.035842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:59405)
0927 22:45:07.036456 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:07.036554 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":326,"threads_started":1}
I20260927 22:45:07.506166 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:07.505288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:39799)
0927 22:45:07.505772 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:07.505867 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260927 22:45:07.735325 18342 raft_consensus.cc:482] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:07.735566 18342 raft_consensus.cc:504] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1fa1097793b47078eb58e42f4345906" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 44139 } } peers { permanent_uuid: "6465cd858a7d439e978b937acf518396" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 44225 } } peers { permanent_uuid: "026402c65d41406ea13c1f24394fbd0e" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 33679 } }
I20260927 22:45:07.736390 18342 leader_election.cc:288] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139), 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679)
W20260927 22:45:07.737428 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1fa1097793b47078eb58e42f4345906 (127.1.53.67:44139): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
W20260927 22:45:07.737712 17935 leader_election.cc:341] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 026402c65d41406ea13c1f24394fbd0e (127.1.53.66:33679): Not found: Tablet not found: 31f6b2916f264ebcbf70a06605a164e2
I20260927 22:45:07.737804 17935 leader_election.cc:302] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6465cd858a7d439e978b937acf518396; no voters: 026402c65d41406ea13c1f24394fbd0e, d1fa1097793b47078eb58e42f4345906
I20260927 22:45:07.738180 18342 raft_consensus.cc:2726] T 31f6b2916f264ebcbf70a06605a164e2 P 6465cd858a7d439e978b937acf518396 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:45:07.981420 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:07.980678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:35399)
0927 22:45:07.981148 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:07.981230 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":248,"threads_started":1}
I20260927 22:45:08.038779 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:08.037921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:55129)
0927 22:45:08.038465 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:08.038562 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260927 22:45:08.507967 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:08.507130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:42319)
0927 22:45:08.507667 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:08.507765 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":268,"threads_started":1}
I20260927 22:45:08.983203 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:08.982470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:50825)
0927 22:45:08.982936 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:08.983041 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260927 22:45:09.040920 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:09.040050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:36335)
0927 22:45:09.040637 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:09.040727 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":358,"threads_started":1}
I20260927 22:45:09.510131 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:09.509187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:47003)
0927 22:45:09.509743 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:09.509874 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":297,"threads_started":1}
I20260927 22:45:09.985107 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:09.984213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:45775)
0927 22:45:09.984826 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:09.984913 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":326,"threads_started":1}
I20260927 22:45:10.042750 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:10.041909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:42021)
0927 22:45:10.042472 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:10.042568 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":356,"threads_started":1}
I20260927 22:45:10.512036 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:10.511220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:49381)
0927 22:45:10.511749 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:10.511847 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":256,"threads_started":1}
I20260927 22:45:10.986963 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:10.986114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:35213)
0927 22:45:10.986669 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:10.986775 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
I20260927 22:45:11.044700 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:11.043805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:35113)
0927 22:45:11.044419 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:11.044513 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":306,"threads_started":1}
I20260927 22:45:11.513813 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:11.513079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:53299)
0927 22:45:11.513546 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:11.513637 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260927 22:45:11.988866 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:11.987986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:34511)
0927 22:45:11.988603 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:11.988692 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":365,"threads_started":1}
I20260927 22:45:12.046603 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:12.045749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:36931)
0927 22:45:12.046312 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:12.046413 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":311,"threads_started":1}
I20260927 22:45:12.515944 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:12.515008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:48835)
0927 22:45:12.515636 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:12.515744 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":318,"threads_started":1}
I20260927 22:45:12.990846 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:12.990032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:33465)
0927 22:45:12.990564 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:12.990674 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":249,"threads_started":1}
I20260927 22:45:13.048646 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:13.047743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:36257)
0927 22:45:13.048347 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:13.048440 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":353,"threads_started":1}
I20260927 22:45:13.517575 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:13.516834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:53091)
0927 22:45:13.517297 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:13.517376 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":270,"threads_started":1}
I20260927 22:45:13.993049 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:13.992056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:54107)
0927 22:45:13.992682 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:13.992790 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":322,"threads_started":1}
I20260927 22:45:14.050908 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:14.050177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.65:43773)
0927 22:45:14.050641 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:14.050738 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":221,"threads_started":1}
I20260927 22:45:14.519496 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:14.518721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.66:45597)
0927 22:45:14.519199 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:14.519314 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":215,"threads_started":1}
I20260927 22:45:14.968091 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 17876
I20260927 22:45:14.984226 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:14.969353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.1.53.65:44225)
0927 22:45:14.969949 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:14.969962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:14.970000 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:45:14.970099 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:14.970109 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:14.970428 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:14.970578 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:14.972052 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:14.973210 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:14.974597 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:14.975225 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:14.975328 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:14.978451 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:14.978465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:14.978481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:14.978517 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:14.981571 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:14.982243 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:14.982250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:14.982252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:14.982408 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:14.982810 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:14.982813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:14.982815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:14.983179 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:14.983407 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:14.983739 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:45:14.983940 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":264,"threads_started":1}
I20260927 22:45:14.985700 18041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:14.995223 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:14.994295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46099 (local address 127.1.53.67:39087)
0927 22:45:14.994848 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:14.994991 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.126:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":255,"threads_started":1}
I20260927 22:45:15.078419 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 17876
I20260927 22:45:15.097126 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18050
I20260927 22:45:15.111609 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:15.097950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.1.53.66:33679)
0927 22:45:15.098577 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:15.098601 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:15.098618 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:45:15.098674 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:15.098678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:15.098737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:15.098824 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:15.100343 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:15.101188 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:15.102429 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:15.102852 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:15.102955 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:15.105783 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:15.105789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:15.105793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:15.105829 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:15.108690 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:15.109380 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:15.109384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:15.109386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:15.109535 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:15.109988 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:15.109992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:15.109995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:15.110349 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:15.110593 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:15.110974 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 22:45:15.111168 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":415,"threads_started":1}
I20260927 22:45:15.112462 18173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:15.198930 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18050
I20260927 22:45:15.217209 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18181
I20260927 22:45:15.232400 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:15.218242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.1.53.67:44139)
0927 22:45:15.218824 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:15.218830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:15.218848 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:45:15.218905 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:15.218909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:15.218991 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:15.219091 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:15.220609 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:15.221461 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:15.222729 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:15.223293 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:15.223393 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:15.226198 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:15.226205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:15.226209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:15.226255 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:15.229674 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:15.230524 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:15.230528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:15.230531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:15.230704 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:15.231102 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:15.231107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:15.231110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:15.231376 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:15.231605 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:15.231847 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:45:15.232021 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":303,"threads_started":1}
I20260927 22:45:15.233201 18304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:15.331471 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18181
2026-09-27T22:45:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64624 ms)
[----------] 30 tests from SecurityITest (302551 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:15 dist-test-slave-pp80 krb5kdc[18374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:15 dist-test-slave-pp80 krb5kdc[18374](info): set up 2 sockets
Sep 27 22:45:15 dist-test-slave-pp80 krb5kdc[18374](info): commencing operation
krb5kdc: starting...
W20260927 22:45:16.472414 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:16 dist-test-slave-pp80 krb5kdc[18374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:45:16.741699 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:38437
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:17.094581 18387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:17.094901 18387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:17.094991 18387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:17.102419 18387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:17.102555 18387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:17.102605 18387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:17.102658 18387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:17.102702 18387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:17.113056 18387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:38437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:17.114425 18387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:17.115746 18387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:17.128791 18392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:17.128834 18395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:17.128880 18393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:17.129408 18387 server_base.cc:1034] running on GCE node
I20260927 22:45:17.130316 18387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:17.132057 18387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:17.133306 18387 hybrid_clock.cc:648] HybridClock initialized: now 1790549117133225 us; error 74 us; skew 500 ppm
I20260927 22:45:17.133706 18387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:17 dist-test-slave-pp80 krb5kdc[18374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549117, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:17.146993 18387 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:45:17.150009 18387 webserver.cc:460] Webserver started at http://127.1.53.126:46465/ using document root <none> and password file <none>
I20260927 22:45:17.151257 18387 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:17.151485 18387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:17.152029 18387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:17.157397 18387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "9d621bc30dd1487eacfcb97b44d63f79"
format_stamp: "Formatted at 2026-09-27 22:45:17 on dist-test-slave-pp80"
server_key: "110886e8f873592a59ad9012e175f103"
server_key_iv: "72bed7002c5df355128f4e27758885df"
server_key_version: "encryptionkey@0"
I20260927 22:45:17.158677 18387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "9d621bc30dd1487eacfcb97b44d63f79"
format_stamp: "Formatted at 2026-09-27 22:45:17 on dist-test-slave-pp80"
server_key: "110886e8f873592a59ad9012e175f103"
server_key_iv: "72bed7002c5df355128f4e27758885df"
server_key_version: "encryptionkey@0"
I20260927 22:45:17.165670 18387 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260927 22:45:17.170554 18402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:17.172859 18387 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:45:17.173113 18387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "9d621bc30dd1487eacfcb97b44d63f79"
format_stamp: "Formatted at 2026-09-27 22:45:17 on dist-test-slave-pp80"
server_key: "110886e8f873592a59ad9012e175f103"
server_key_iv: "72bed7002c5df355128f4e27758885df"
server_key_version: "encryptionkey@0"
I20260927 22:45:17.173403 18387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:17.202684 18387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:17.207639 18387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:17.207916 18387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:17.232043 18387 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:38437
I20260927 22:45:17.232061 18453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:38437 every 8 connection(s)
I20260927 22:45:17.234257 18387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:17.238286 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18387
I20260927 22:45:17.238552 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:17.238803 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:17.239461 1237 external_mini_cluster.cc:1428] Setting key 3b22acc2d25973007387ba38cb5fdb29
I20260927 22:45:17.254307 18454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79: Bootstrap starting.
Sep 27 22:45:17 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:17.258369 18454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:17.259876 18454 log.cc:824] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:17.263955 18454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79: No bootstrap required, opened a new log
I20260927 22:45:17.266697 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:17.241769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.1.53.126:38437)
0927 22:45:17.242648 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:17.242664 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:17.242717 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:45:17.244623 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:17.244655 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:17.245222 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:17.245710 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:17.247155 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:17.249090 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:17.250855 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:17.252048 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:17.252341 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:17.257550 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:17.257563 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:17.257584 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:17.257638 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:17.261948 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:17.262954 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:17.262961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:17.262968 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:17.263185 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:17.263649 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:17.263652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:17.263654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:17.264356 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:17.264616 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:17.265114 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 22:45:17.265560 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":353,"threads_started":1}
I20260927 22:45:17.273137 18454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d621bc30dd1487eacfcb97b44d63f79" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38437 } }
I20260927 22:45:17.273588 18454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:17.273703 18454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d621bc30dd1487eacfcb97b44d63f79, State: Initialized, Role: FOLLOWER
I20260927 22:45:17.274322 18454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d621bc30dd1487eacfcb97b44d63f79" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38437 } }
I20260927 22:45:17.274612 18454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:17.274765 18454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:17.274955 18454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:17.277568 18454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d621bc30dd1487eacfcb97b44d63f79" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38437 } }
I20260927 22:45:17.278194 18454 leader_election.cc:302] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d621bc30dd1487eacfcb97b44d63f79; no voters:
I20260927 22:45:17.278827 18454 leader_election.cc:288] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:17.278999 18459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:17.280145 18459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [term 1 LEADER]: Becoming Leader. State: Replica: 9d621bc30dd1487eacfcb97b44d63f79, State: Running, Role: LEADER
I20260927 22:45:17.280748 18459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d621bc30dd1487eacfcb97b44d63f79" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38437 } }
I20260927 22:45:17.281140 18454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:17.284677 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d621bc30dd1487eacfcb97b44d63f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d621bc30dd1487eacfcb97b44d63f79" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38437 } } }
I20260927 22:45:17.285210 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:17.284287 18461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d621bc30dd1487eacfcb97b44d63f79. Latest consensus state: current_term: 1 leader_uuid: "9d621bc30dd1487eacfcb97b44d63f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d621bc30dd1487eacfcb97b44d63f79" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38437 } } }
I20260927 22:45:17.285662 18461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:17.286336 18468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:17.292589 18468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:17.302578 18468 catalog_manager.cc:1347] Generated new cluster ID: e2d7a64ff2a949999f6ee3bfb27eea0a
I20260927 22:45:17.302726 18468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:17.328543 18468 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:17.330106 18468 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:17.342013 18468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79: Generated new TSK 0
I20260927 22:45:17.343039 18468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:45:17.408473 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:38437
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:17.764060 18480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:17.764400 18480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:17.764572 18480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:17.772285 18480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:17.772433 18480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:17.772617 18480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:45:17.783010 18480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:17.784538 18480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:17.785898 18480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:17.800791 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:17.800949 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:17.801424 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:17.802359 18480 server_base.cc:1034] running on GCE node
I20260927 22:45:17.803052 18480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:17.804104 18480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:17.805330 18480 hybrid_clock.cc:648] HybridClock initialized: now 1790549117805271 us; error 55 us; skew 500 ppm
I20260927 22:45:17.805716 18480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:17 dist-test-slave-pp80 krb5kdc[18374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549117, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:17.809312 18480 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:45:17.810928 18480 webserver.cc:460] Webserver started at http://127.1.53.65:41257/ using document root <none> and password file <none>
I20260927 22:45:17.811685 18480 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:17.811838 18480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:17.812173 18480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:17.815423 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "0cbff1bdf7554e0cae388dceb5ab1f11"
format_stamp: "Formatted at 2026-09-27 22:45:17 on dist-test-slave-pp80"
server_key: "309ac815aa04f0765e2f67d751aad78d"
server_key_iv: "b6f7fc8388ad9b64017ded841547cb16"
server_key_version: "encryptionkey@0"
I20260927 22:45:17.816357 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "0cbff1bdf7554e0cae388dceb5ab1f11"
format_stamp: "Formatted at 2026-09-27 22:45:17 on dist-test-slave-pp80"
server_key: "309ac815aa04f0765e2f67d751aad78d"
server_key_iv: "b6f7fc8388ad9b64017ded841547cb16"
server_key_version: "encryptionkey@0"
I20260927 22:45:17.822220 18480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:45:17.826207 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:17.827751 18480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:45:17.827963 18480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "0cbff1bdf7554e0cae388dceb5ab1f11"
format_stamp: "Formatted at 2026-09-27 22:45:17 on dist-test-slave-pp80"
server_key: "309ac815aa04f0765e2f67d751aad78d"
server_key_iv: "b6f7fc8388ad9b64017ded841547cb16"
server_key_version: "encryptionkey@0"
I20260927 22:45:17.828217 18480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:17.846551 18480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:17.851747 18480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:17.852092 18480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:17.853248 18480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:17.854993 18480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:17.855109 18480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:17.855274 18480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:17.855360 18480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:17.893754 18480 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:38665
I20260927 22:45:17.893810 18607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:38665 every 8 connection(s)
I20260927 22:45:17.895219 18480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:45:17.904909 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18480
I20260927 22:45:17.905112 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:45:17.905735 1237 external_mini_cluster.cc:1428] Setting key 1ab0e23f802eda5c74054dfd7b80fda7
Sep 27 22:45:17 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549117, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:17.918520 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:17.898128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:47597 (local address 127.1.53.126:38437)
0927 22:45:17.898593 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:17.898599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:17.899699 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 22:45:17.901724 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:17.901729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:17.901807 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:17.901910 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:17.903551 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:17.904374 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:17.906248 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:17.906791 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:17.906926 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:17.911092 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:17.911102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:17.911109 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:17.911161 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:17.914897 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:17.915798 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:17.915807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:17.915814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:17.915978 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:17.916568 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:17.916572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:17.916574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:17.916845 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:17.916969 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:17.918077 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0927 22:45:17.918265 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260927 22:45:17.920065 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:17.898696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:38437 (local address 127.1.53.65:47597)
0927 22:45:17.899508 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:17.899558 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:17.901438 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:17.902194 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:17.902205 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:17.902767 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:17.903369 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:17.903386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:17.904545 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:17.904550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:17.906088 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:17.906101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:17.907106 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:17.907115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:17.907413 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:17.908433 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:17.908457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:17.910796 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:17.915144 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:17.915152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:17.915169 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:17.915660 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:17.916124 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:17.916128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:17.916131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:17.916435 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:17.916976 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:17.916986 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:17.917535 (+ 549us) client_negotiation.cc:769] Sending connection context
0927 22:45:17.917890 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 22:45:17.918523 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":347,"threads_started":1}
I20260927 22:45:17.921917 18608 heartbeater.cc:347] Connected to a master server at 127.1.53.126:38437
I20260927 22:45:17.922338 18608 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:17.923175 18608 heartbeater.cc:511] Master 127.1.53.126:38437 requested a full tablet report, sending...
I20260927 22:45:17.925629 18419 ts_manager.cc:194] Registered new tserver with Master: 0cbff1bdf7554e0cae388dceb5ab1f11 (127.1.53.65:38665)
I20260927 22:45:17.927299 18419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:47597
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:45:17.964764 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:38437
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:18.311602 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:18.311902 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:18.312152 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:18.319932 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:18.320063 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:18.320326 18614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:45:18.330582 18614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:18.331970 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:18.333526 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:18.347529 18619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:18.347492 18620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:18.347438 18622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:18.348688 18614 server_base.cc:1034] running on GCE node
I20260927 22:45:18.349378 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:18.350371 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:18.351577 18614 hybrid_clock.cc:648] HybridClock initialized: now 1790549118351501 us; error 64 us; skew 500 ppm
I20260927 22:45:18.351879 18614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:18 dist-test-slave-pp80 krb5kdc[18374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:18.355260 18614 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:45:18.356863 18614 webserver.cc:460] Webserver started at http://127.1.53.66:40861/ using document root <none> and password file <none>
I20260927 22:45:18.357549 18614 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:18.357647 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:18.357920 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:18.360949 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "6f462872a18540b780bf0c857620eaeb"
format_stamp: "Formatted at 2026-09-27 22:45:18 on dist-test-slave-pp80"
server_key: "5a3a28057621a8f3f1c10b15d2806632"
server_key_iv: "9d568e978db1835eb4e5ea401f53b857"
server_key_version: "encryptionkey@0"
I20260927 22:45:18.361708 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "6f462872a18540b780bf0c857620eaeb"
format_stamp: "Formatted at 2026-09-27 22:45:18 on dist-test-slave-pp80"
server_key: "5a3a28057621a8f3f1c10b15d2806632"
server_key_iv: "9d568e978db1835eb4e5ea401f53b857"
server_key_version: "encryptionkey@0"
I20260927 22:45:18.367086 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:45:18.370921 18629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:18.372740 18614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:45:18.372893 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "6f462872a18540b780bf0c857620eaeb"
format_stamp: "Formatted at 2026-09-27 22:45:18 on dist-test-slave-pp80"
server_key: "5a3a28057621a8f3f1c10b15d2806632"
server_key_iv: "9d568e978db1835eb4e5ea401f53b857"
server_key_version: "encryptionkey@0"
I20260927 22:45:18.373086 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:18.402951 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:18.407733 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:18.408016 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:18.409058 18614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:18.410429 18614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:18.410503 18614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:18.410589 18614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:18.410633 18614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:18.445837 18614 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:36167
I20260927 22:45:18.445890 18741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:36167 every 8 connection(s)
I20260927 22:45:18.447134 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:45:18.450628 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18614
I20260927 22:45:18.450803 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:45:18.451244 1237 external_mini_cluster.cc:1428] Setting key 7010022f5c0b82d9dbeb213ff8aa4c18
Sep 27 22:45:18 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:18.470702 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:18.449899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:42141 (local address 127.1.53.126:38437)
0927 22:45:18.450444 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:18.450451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:18.451643 (+ 1192us) server_negotiation.cc:407] Connection header received
0927 22:45:18.453612 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:18.453617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:18.453708 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:18.453829 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:18.455363 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.456552 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.458197 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.458965 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.459110 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:18.463356 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:18.463362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:18.463366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:18.463405 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:18.466743 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:18.467841 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:18.467845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:18.467848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:18.468053 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:18.468692 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:18.468696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:18.468699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:18.469014 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:18.469152 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:18.470191 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0927 22:45:18.470390 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":235,"threads_started":1}
I20260927 22:45:18.472283 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:18.450550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:38437 (local address 127.1.53.66:42141)
0927 22:45:18.451438 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:18.451498 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:18.453340 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:18.454024 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:18.454033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:18.454582 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:18.455178 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:18.455197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.456702 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.456706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:18.458044 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:18.458058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.459234 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.459239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:18.459476 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:18.460287 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:18.460333 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:18.463046 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:18.466932 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:18.466946 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:18.466968 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:18.467684 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:18.468187 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:18.468195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:18.468201 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:18.468508 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:18.469164 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:18.469176 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:18.469680 (+ 504us) client_negotiation.cc:769] Sending connection context
0927 22:45:18.470024 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 22:45:18.470667 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":420,"threads_started":1}
I20260927 22:45:18.473958 18742 heartbeater.cc:347] Connected to a master server at 127.1.53.126:38437
I20260927 22:45:18.474350 18742 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:18.475162 18742 heartbeater.cc:511] Master 127.1.53.126:38437 requested a full tablet report, sending...
I20260927 22:45:18.476907 18419 ts_manager.cc:194] Registered new tserver with Master: 6f462872a18540b780bf0c857620eaeb (127.1.53.66:36167)
I20260927 22:45:18.477779 18419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:42141
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:45:18.512780 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:38437
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:18.859858 18748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:18.860162 18748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:18.860314 18748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:18.867805 18748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:18.867935 18748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:18.868052 18748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:45:18.878981 18748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:18.880389 18748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:18.881865 18748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:18.895875 18754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:18.895875 18753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:18.896708 18756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:18.900651 18748 server_base.cc:1034] running on GCE node
I20260927 22:45:18.901307 18748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:18.902370 18748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:18.903594 18748 hybrid_clock.cc:648] HybridClock initialized: now 1790549118903547 us; error 40 us; skew 500 ppm
I20260927 22:45:18.903908 18748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:18 dist-test-slave-pp80 krb5kdc[18374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:18.907501 18748 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:45:18.909258 18748 webserver.cc:460] Webserver started at http://127.1.53.67:33763/ using document root <none> and password file <none>
I20260927 22:45:18.910106 18748 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:18.910215 18748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:18.910496 18748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:18.913740 18748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "c7a0c27135f6400ba0b4475a7fec69f4"
format_stamp: "Formatted at 2026-09-27 22:45:18 on dist-test-slave-pp80"
server_key: "03bc5ff9e0c9be4923953a276d5f3f7f"
server_key_iv: "fe9cb0f49f6c9daab07fa366da7954c0"
server_key_version: "encryptionkey@0"
I20260927 22:45:18.914556 18748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "c7a0c27135f6400ba0b4475a7fec69f4"
format_stamp: "Formatted at 2026-09-27 22:45:18 on dist-test-slave-pp80"
server_key: "03bc5ff9e0c9be4923953a276d5f3f7f"
server_key_iv: "fe9cb0f49f6c9daab07fa366da7954c0"
server_key_version: "encryptionkey@0"
I20260927 22:45:18.920392 18748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:45:18.924587 18763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:18.926072 18748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:18.926314 18748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "c7a0c27135f6400ba0b4475a7fec69f4"
format_stamp: "Formatted at 2026-09-27 22:45:18 on dist-test-slave-pp80"
server_key: "03bc5ff9e0c9be4923953a276d5f3f7f"
server_key_iv: "fe9cb0f49f6c9daab07fa366da7954c0"
server_key_version: "encryptionkey@0"
I20260927 22:45:18.926533 18748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:18.930635 18608 heartbeater.cc:503] Master 127.1.53.126:38437 was elected leader, sending a full tablet report...
I20260927 22:45:18.939652 18748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:18.944352 18748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:18.944679 18748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:18.945734 18748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:18.947149 18748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:18.947249 18748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:18.947386 18748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:18.947458 18748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:18.982846 18748 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:42767
I20260927 22:45:18.982892 18875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:42767 every 8 connection(s)
I20260927 22:45:18.984169 18748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:45:18.991194 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18748
I20260927 22:45:18.991423 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:45:18.991894 1237 external_mini_cluster.cc:1428] Setting key 299675d3cae3946309bf100d47751555
Sep 27 22:45:18 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:19.005632 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:18.986895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:36269 (local address 127.1.53.126:38437)
0927 22:45:18.987459 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:18.987466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:18.988554 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 22:45:18.990402 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:18.990407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:18.990502 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:18.990614 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:18.992300 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.993262 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.995413 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.995811 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.995908 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:18.999606 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:18.999611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:18.999614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:18.999649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:19.002387 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.003163 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.003169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.003173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.003300 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.003739 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.003745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.003748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.004033 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:19.004201 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.004933 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.005172 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":298,"threads_started":1}
I20260927 22:45:19.006846 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:18.987606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:38437 (local address 127.1.53.67:36269)
0927 22:45:18.988382 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:18.988428 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:18.990112 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:18.990841 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:18.990850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:18.991448 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:18.992100 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:18.992121 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.993480 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.993488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:18.995290 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:18.995302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:18.996045 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:18.996052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:18.996349 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:18.997043 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:18.997067 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:18.999378 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:19.002553 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:19.002563 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:19.002583 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:19.003029 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:19.003411 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:19.003416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:19.003420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:19.003627 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:19.004165 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:19.004175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:19.004564 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 22:45:19.004838 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 22:45:19.005423 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":280,"threads_started":1}
I20260927 22:45:19.008447 18876 heartbeater.cc:347] Connected to a master server at 127.1.53.126:38437
I20260927 22:45:19.008749 18876 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:19.009431 18876 heartbeater.cc:511] Master 127.1.53.126:38437 requested a full tablet report, sending...
I20260927 22:45:19.010869 18418 ts_manager.cc:194] Registered new tserver with Master: c7a0c27135f6400ba0b4475a7fec69f4 (127.1.53.67:42767)
I20260927 22:45:19.011825 18418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:36269
I20260927 22:45:19.021098 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:45:19.040975 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.026318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.1.53.126:38437)
0927 22:45:19.026656 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.026663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.026896 (+ 233us) server_negotiation.cc:407] Connection header received
0927 22:45:19.027065 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.027069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.027140 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.027236 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:19.028989 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.029921 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.031574 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.032042 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.032181 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.035108 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:19.035114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:19.035118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:19.035159 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:19.038010 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.038775 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.038779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.038781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.038916 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.039388 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.039391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.039393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.039778 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:19.039964 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.040417 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.040606 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260927 22:45:19.045550 18418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:45:19.053082 18418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:45:19.096184 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.067030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38665 (local address 127.0.0.1:59052)
0927 22:45:19.085065 (+ 18035us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:19.085091 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:19.085288 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:19.086635 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:19.086648 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:19.086692 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:19.087263 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.087280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.089354 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.089359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.092879 (+ 3520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.092895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.094799 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.094805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.095283 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.095352 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:45:19.095519 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:45:19.095732 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17861,"thread_start_us":21361,"threads_started":1}
I20260927 22:45:19.096207 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.067369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.1.53.65:38665)
0927 22:45:19.084282 (+ 16913us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.084297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.085448 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 22:45:19.085603 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.085616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.086018 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.086275 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:19.087483 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.089182 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.093074 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.094602 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.094738 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.095011 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.095680 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.095868 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16612,"thread_start_us":16956,"threads_started":1}
I20260927 22:45:19.097901 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.068048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.1.53.67:42767)
0927 22:45:19.087703 (+ 19655us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.087714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.088980 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 22:45:19.089080 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.089089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.089378 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.090169 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:19.090648 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.092299 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.095100 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.096754 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.096882 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.097120 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.097375 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.097558 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19448,"thread_start_us":19674,"threads_started":1}
I20260927 22:45:19.098268 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.067406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:42767 (local address 127.0.0.1:60746)
0927 22:45:19.088702 (+ 21296us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:19.088722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:19.088842 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:19.089642 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:19.089648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:19.089673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:19.090325 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.090340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.092489 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.092497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.094903 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.094917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.096994 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.097002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.097179 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.097213 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:45:19.097318 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:45:19.097473 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21121,"spinlock_wait_cycles":2304,"thread_start_us":17317,"threads_started":1}
I20260927 22:45:19.098901 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.067141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.1.53.66:36167)
0927 22:45:19.088976 (+ 21835us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.088986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.089016 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:45:19.089138 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.089147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.089437 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.090154 (+ 717us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:19.090863 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.092434 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.094573 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.097322 (+ 2749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.097458 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.097640 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.098390 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.098578 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21053,"thread_start_us":20936,"threads_started":1}
I20260927 22:45:19.099522 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.066764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:36167 (local address 127.0.0.1:53176)
0927 22:45:19.079805 (+ 13041us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:19.079887 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:19.080194 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:19.089710 (+ 9516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:19.089715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:19.089765 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:19.090679 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.090695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.092586 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.092592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.094386 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.094402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.097800 (+ 3398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.097807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.098007 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.098043 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:45:19.098139 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:45:19.098271 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12596,"spinlock_wait_cycles":1280,"thread_start_us":12357,"threads_started":1}
I20260927 22:45:19.101096 18543 tablet_service.cc:1467] Processing CreateTablet for tablet 1262a7ffef6e49cbae700dd7b62d45e0 (DEFAULT_TABLE table=test-table [id=5892aaf428444bb49d2e33a292626dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:19.101910 18811 tablet_service.cc:1467] Processing CreateTablet for tablet 1262a7ffef6e49cbae700dd7b62d45e0 (DEFAULT_TABLE table=test-table [id=5892aaf428444bb49d2e33a292626dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:19.102799 18543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1262a7ffef6e49cbae700dd7b62d45e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:19.103123 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1262a7ffef6e49cbae700dd7b62d45e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:19.103734 18677 tablet_service.cc:1467] Processing CreateTablet for tablet 1262a7ffef6e49cbae700dd7b62d45e0 (DEFAULT_TABLE table=test-table [id=5892aaf428444bb49d2e33a292626dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:19.105914 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1262a7ffef6e49cbae700dd7b62d45e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:19.114603 18895 tablet_bootstrap.cc:492] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Bootstrap starting.
I20260927 22:45:19.117920 18895 tablet_bootstrap.cc:654] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:19.119006 18896 tablet_bootstrap.cc:492] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Bootstrap starting.
I20260927 22:45:19.120062 18895 log.cc:824] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:19.122907 18896 tablet_bootstrap.cc:654] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:19.122869 18898 tablet_bootstrap.cc:492] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Bootstrap starting.
I20260927 22:45:19.123626 18895 tablet_bootstrap.cc:492] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: No bootstrap required, opened a new log
I20260927 22:45:19.123917 18895 ts_tablet_manager.cc:1397] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 22:45:19.124557 18896 log.cc:824] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:19.126761 18898 tablet_bootstrap.cc:654] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:19.127830 18896 tablet_bootstrap.cc:492] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: No bootstrap required, opened a new log
I20260927 22:45:19.127980 18898 log.cc:824] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:19.128111 18896 ts_tablet_manager.cc:1397] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 22:45:19.134866 18895 raft_consensus.cc:348] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.135423 18895 raft_consensus.cc:374] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:19.135530 18895 raft_consensus.cc:729] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a0c27135f6400ba0b4475a7fec69f4, State: Initialized, Role: FOLLOWER
I20260927 22:45:19.136264 18895 consensus_queue.cc:260] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.137755 18876 heartbeater.cc:503] Master 127.1.53.126:38437 was elected leader, sending a full tablet report...
I20260927 22:45:19.138434 18895 ts_tablet_manager.cc:1428] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 22:45:19.138518 18898 tablet_bootstrap.cc:492] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: No bootstrap required, opened a new log
I20260927 22:45:19.139416 18898 ts_tablet_manager.cc:1397] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260927 22:45:19.139196 18896 raft_consensus.cc:348] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.139719 18896 raft_consensus.cc:374] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:19.139827 18896 raft_consensus.cc:729] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbff1bdf7554e0cae388dceb5ab1f11, State: Initialized, Role: FOLLOWER
I20260927 22:45:19.140619 18896 consensus_queue.cc:260] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.142953 18896 ts_tablet_manager.cc:1428] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260927 22:45:19.147881 18609 tablet.cc:2369] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:45:19.150333 18898 raft_consensus.cc:348] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.150765 18898 raft_consensus.cc:374] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:19.150848 18898 raft_consensus.cc:729] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f462872a18540b780bf0c857620eaeb, State: Initialized, Role: FOLLOWER
I20260927 22:45:19.151533 18898 consensus_queue.cc:260] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.152750 18742 heartbeater.cc:503] Master 127.1.53.126:38437 was elected leader, sending a full tablet report...
I20260927 22:45:19.153599 18898 ts_tablet_manager.cc:1428] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260927 22:45:19.199334 18743 tablet.cc:2369] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:45:19.235865 18877 tablet.cc:2369] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:45:19.246031 18903 raft_consensus.cc:482] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:19.246390 18903 raft_consensus.cc:504] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.248195 18903 leader_election.cc:288] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cbff1bdf7554e0cae388dceb5ab1f11 (127.1.53.65:38665), c7a0c27135f6400ba0b4475a7fec69f4 (127.1.53.67:42767)
I20260927 22:45:19.255657 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.248664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:60739 (local address 127.1.53.65:38665)
0927 22:45:19.248885 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.248892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.249672 (+ 780us) server_negotiation.cc:407] Connection header received
0927 22:45:19.249841 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.249848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.249935 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.250040 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:19.250764 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.251639 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.253568 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.254541 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.254657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.254800 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.255204 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.255357 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:45:19.255657 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.248816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:39133 (local address 127.1.53.67:42767)
0927 22:45:19.249094 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.249103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.249869 (+ 766us) server_negotiation.cc:407] Connection header received
0927 22:45:19.250125 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.250131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.250215 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.250330 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:19.250974 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.251681 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.253756 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.254598 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.254767 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.254892 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.255277 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.255419 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:45:19.255777 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.248394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38665 (local address 127.1.53.66:60739)
0927 22:45:19.249525 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:19.249550 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:19.249688 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:19.250106 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:19.250111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:19.250141 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:19.250593 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.250605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.251784 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.251791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.253391 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.253404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.254799 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.254806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.255025 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.255053 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:45:19.255133 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:45:19.255267 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":29,"thread_start_us":680,"threads_started":1}
I20260927 22:45:19.256052 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.248687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:42767 (local address 127.1.53.66:39133)
0927 22:45:19.249747 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:19.249769 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:19.249910 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:19.250358 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:19.250362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:19.250383 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:19.250800 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.250815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.251837 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.251841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.253604 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:19.253619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.254865 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.254871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:19.255085 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.255133 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:45:19.255237 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:45:19.255365 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":38,"thread_start_us":682,"threads_started":1}
I20260927 22:45:19.256983 18563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1262a7ffef6e49cbae700dd7b62d45e0" candidate_uuid: "6f462872a18540b780bf0c857620eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" is_pre_election: true
I20260927 22:45:19.257193 18831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1262a7ffef6e49cbae700dd7b62d45e0" candidate_uuid: "6f462872a18540b780bf0c857620eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" is_pre_election: true
I20260927 22:45:19.257685 18831 raft_consensus.cc:2454] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f462872a18540b780bf0c857620eaeb in term 0.
I20260927 22:45:19.257686 18563 raft_consensus.cc:2454] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f462872a18540b780bf0c857620eaeb in term 0.
I20260927 22:45:19.258474 18630 leader_election.cc:302] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cbff1bdf7554e0cae388dceb5ab1f11, 6f462872a18540b780bf0c857620eaeb, c7a0c27135f6400ba0b4475a7fec69f4; no voters:
I20260927 22:45:19.258906 18903 raft_consensus.cc:2781] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:45:19.259080 18903 raft_consensus.cc:482] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:45:19.259193 18903 raft_consensus.cc:3037] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:19.261533 18903 raft_consensus.cc:504] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.262197 18903 leader_election.cc:288] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [CANDIDATE]: Term 1 election: Requested vote from peers 0cbff1bdf7554e0cae388dceb5ab1f11 (127.1.53.65:38665), c7a0c27135f6400ba0b4475a7fec69f4 (127.1.53.67:42767)
I20260927 22:45:19.262781 18831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1262a7ffef6e49cbae700dd7b62d45e0" candidate_uuid: "6f462872a18540b780bf0c857620eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a0c27135f6400ba0b4475a7fec69f4"
I20260927 22:45:19.262777 18563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1262a7ffef6e49cbae700dd7b62d45e0" candidate_uuid: "6f462872a18540b780bf0c857620eaeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11"
I20260927 22:45:19.263006 18831 raft_consensus.cc:3037] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:19.263026 18563 raft_consensus.cc:3037] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:19.265394 18831 raft_consensus.cc:2454] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f462872a18540b780bf0c857620eaeb in term 1.
I20260927 22:45:19.265657 18563 raft_consensus.cc:2454] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f462872a18540b780bf0c857620eaeb in term 1.
I20260927 22:45:19.265970 18633 leader_election.cc:302] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f462872a18540b780bf0c857620eaeb, c7a0c27135f6400ba0b4475a7fec69f4; no voters:
I20260927 22:45:19.266319 18903 raft_consensus.cc:2781] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:19.266794 18903 raft_consensus.cc:686] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 1 LEADER]: Becoming Leader. State: Replica: 6f462872a18540b780bf0c857620eaeb, State: Running, Role: LEADER
I20260927 22:45:19.267371 18903 consensus_queue.cc:237] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } }
I20260927 22:45:19.271989 18418 catalog_manager.cc:5495] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f462872a18540b780bf0c857620eaeb (127.1.53.66). New cstate: current_term: 1 leader_uuid: "6f462872a18540b780bf0c857620eaeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f462872a18540b780bf0c857620eaeb" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 36167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 } health_report { overall_health: UNKNOWN } } }
I20260927 22:45:19.294767 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.288609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.1.53.66:36167)
0927 22:45:19.288851 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.288860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.288950 (+ 90us) server_negotiation.cc:407] Connection header received
0927 22:45:19.289217 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.289221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.289304 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.289473 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:45:19.289913 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.290796 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.292437 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.292897 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.293038 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.293579 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:45:19.294116 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:45:19.294284 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.294372 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.294459 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:45:19.304122 18563 raft_consensus.cc:1261] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 1 FOLLOWER]: Refusing update from remote peer 6f462872a18540b780bf0c857620eaeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:19.304122 18831 raft_consensus.cc:1261] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 1 FOLLOWER]: Refusing update from remote peer 6f462872a18540b780bf0c857620eaeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:19.305361 18906 consensus_queue.cc:1035] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cbff1bdf7554e0cae388dceb5ab1f11" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:19.305761 18903 consensus_queue.cc:1035] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a0c27135f6400ba0b4475a7fec69f4" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 42767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:19.317305 18911 mvcc.cc:205] Tried to move back new op lower bound from 7334089192658092032 to 7334089192523251712. Current Snapshot: MvccSnapshot[applied={T|T < 7334089192658092032}]
I20260927 22:45:19.334209 18418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:19.334535 18418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:19.338732 18418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d621bc30dd1487eacfcb97b44d63f79: Sending DeleteTablet for 3 replicas of tablet 1262a7ffef6e49cbae700dd7b62d45e0
I20260927 22:45:19.340096 18543 tablet_service.cc:1514] Processing DeleteTablet for tablet 1262a7ffef6e49cbae700dd7b62d45e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:19 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:59052
I20260927 22:45:19.340143 18677 tablet_service.cc:1514] Processing DeleteTablet for tablet 1262a7ffef6e49cbae700dd7b62d45e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:19 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:53176
I20260927 22:45:19.341104 18918 tablet_replica.cc:330] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: stopping tablet replica
I20260927 22:45:19.341156 18811 tablet_service.cc:1514] Processing DeleteTablet for tablet 1262a7ffef6e49cbae700dd7b62d45e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:19 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:60746
I20260927 22:45:19.341593 18918 raft_consensus.cc:2229] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:19.342139 18918 raft_consensus.cc:2258] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:19.342993 18919 tablet_replica.cc:330] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: stopping tablet replica
I20260927 22:45:19.343191 18920 tablet_replica.cc:330] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: stopping tablet replica
I20260927 22:45:19.343429 18919 raft_consensus.cc:2229] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:45:19.343575 18920 raft_consensus.cc:2229] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:19.344100 18920 raft_consensus.cc:2258] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:19.344133 18919 raft_consensus.cc:2258] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:19.344259 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18480
I20260927 22:45:19.346354 18920 ts_tablet_manager.cc:1905] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:19.346464 18918 ts_tablet_manager.cc:1905] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:19.350375 18919 ts_tablet_manager.cc:1905] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:19.355538 18920 ts_tablet_manager.cc:1918] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:19.355741 18920 log.cc:1196] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/1262a7ffef6e49cbae700dd7b62d45e0
I20260927 22:45:19.356328 18920 ts_tablet_manager.cc:1939] T 1262a7ffef6e49cbae700dd7b62d45e0 P c7a0c27135f6400ba0b4475a7fec69f4: Deleting consensus metadata
I20260927 22:45:19.358417 18406 catalog_manager.cc:4841] TS c7a0c27135f6400ba0b4475a7fec69f4 (127.1.53.67:42767): tablet 1262a7ffef6e49cbae700dd7b62d45e0 (table test-table [id=5892aaf428444bb49d2e33a292626dde]) successfully deleted
I20260927 22:45:19.361016 18919 ts_tablet_manager.cc:1918] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:19.361213 18919 log.cc:1196] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/1262a7ffef6e49cbae700dd7b62d45e0
I20260927 22:45:19.361805 18919 ts_tablet_manager.cc:1939] T 1262a7ffef6e49cbae700dd7b62d45e0 P 6f462872a18540b780bf0c857620eaeb: Deleting consensus metadata
I20260927 22:45:19.363689 18404 catalog_manager.cc:4841] TS 6f462872a18540b780bf0c857620eaeb (127.1.53.66:36167): tablet 1262a7ffef6e49cbae700dd7b62d45e0 (table test-table [id=5892aaf428444bb49d2e33a292626dde]) successfully deleted
I20260927 22:45:19.364632 18918 ts_tablet_manager.cc:1918] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:19.364825 18918 log.cc:1196] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/1262a7ffef6e49cbae700dd7b62d45e0
I20260927 22:45:19.365449 18918 ts_tablet_manager.cc:1939] T 1262a7ffef6e49cbae700dd7b62d45e0 P 0cbff1bdf7554e0cae388dceb5ab1f11: Deleting consensus metadata
Sep 27 22:45:19 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:45:19.367458 18403 catalog_manager.cc:4841] TS 0cbff1bdf7554e0cae388dceb5ab1f11 (127.1.53.65:38665): tablet 1262a7ffef6e49cbae700dd7b62d45e0 (table test-table [id=5892aaf428444bb49d2e33a292626dde]) successfully deleted
I20260927 22:45:19.373708 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.346382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.1.53.65:38665)
0927 22:45:19.346572 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.346579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.356516 (+ 9937us) server_negotiation.cc:407] Connection header received
0927 22:45:19.356875 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.356881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.356974 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.357051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:19.359024 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.359784 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.361845 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.362531 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.362636 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.367534 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:19.367545 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:19.367559 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:19.367614 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:19.370740 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.371466 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.371473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.371476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.371607 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.372010 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.372014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.372017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.372474 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:19.372776 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.373091 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.373349 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:45:19.374437 18603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:19.479390 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18480
I20260927 22:45:19.498826 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18614
Sep 27 22:45:19 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:45:19.518044 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.499611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.1.53.66:36167)
0927 22:45:19.499824 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.499829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.499848 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:45:19.500039 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.500043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.500118 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.500208 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:19.501860 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.502724 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.504713 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.505217 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.505331 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.510733 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:19.510751 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:19.510769 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:19.510827 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:19.514827 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.515691 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.515700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.515703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.515861 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.516329 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.516333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.516335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.516733 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:19.517146 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.517389 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.517631 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 22:45:19.518857 18737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:19.638885 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18614
I20260927 22:45:19.659885 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18748
Sep 27 22:45:19 dist-test-slave-pp80 krb5kdc[18374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:45:19.677577 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:19.660808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.1.53.67:42767)
0927 22:45:19.661043 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:19.661051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:19.661210 (+ 159us) server_negotiation.cc:407] Connection header received
0927 22:45:19.661459 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:19.661480 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:19.661587 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:19.661713 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:19.663102 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.663919 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.665529 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:19.665997 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:19.666153 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:19.671250 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:19.671267 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:19.671295 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:19.671331 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:19.674677 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.675405 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.675413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.675415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.675556 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:19.675918 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:19.675921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:19.675923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:19.676376 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:19.676594 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:19.677010 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:45:19.677178 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 22:45:19.678325 18871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:19.770154 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18748
I20260927 22:45:19.787413 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18387
I20260927 22:45:19.788631 18449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:19.902499 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18387
2026-09-27T22:45:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4572 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:19Z Disabled control of system clock
I20260927 22:45:19.980564 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37609
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:45:20.333824 18929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:20.334157 18929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:20.334239 18929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:20.342022 18929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:20.342156 18929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:20.342195 18929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:20.342244 18929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:20.342289 18929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:20.351725 18929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:20.353193 18929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:20.354938 18929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:20.366955 18937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:20.367125 18934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:20.367132 18935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:20.368076 18929 server_base.cc:1034] running on GCE node
I20260927 22:45:20.369001 18929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:20.370500 18929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:20.371781 18929 hybrid_clock.cc:648] HybridClock initialized: now 1790549120371719 us; error 52 us; skew 500 ppm
I20260927 22:45:20.372225 18929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:20.375010 18929 webserver.cc:460] Webserver started at http://127.1.53.126:35247/ using document root <none> and password file <none>
I20260927 22:45:20.375739 18929 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:20.375885 18929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:20.376302 18929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:20.379305 18929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "3588e69643bd47569b9b7d619a7e7be0"
format_stamp: "Formatted at 2026-09-27 22:45:20 on dist-test-slave-pp80"
server_key: "77df127ab35529f2186f05f76a626985"
server_key_iv: "26dbdd95632e9451354537de62d23993"
server_key_version: "encryptionkey@0"
I20260927 22:45:20.380167 18929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "3588e69643bd47569b9b7d619a7e7be0"
format_stamp: "Formatted at 2026-09-27 22:45:20 on dist-test-slave-pp80"
server_key: "77df127ab35529f2186f05f76a626985"
server_key_iv: "26dbdd95632e9451354537de62d23993"
server_key_version: "encryptionkey@0"
I20260927 22:45:20.385555 18929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:45:20.389178 18943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:20.390641 18929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:20.390790 18929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "3588e69643bd47569b9b7d619a7e7be0"
format_stamp: "Formatted at 2026-09-27 22:45:20 on dist-test-slave-pp80"
server_key: "77df127ab35529f2186f05f76a626985"
server_key_iv: "26dbdd95632e9451354537de62d23993"
server_key_version: "encryptionkey@0"
I20260927 22:45:20.390982 18929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:20.426766 18929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:20.427713 18929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:20.427999 18929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:20.448988 18929 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37609
I20260927 22:45:20.449034 18994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37609 every 8 connection(s)
I20260927 22:45:20.450510 18929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:20.454447 18995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:20.456691 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 18929
I20260927 22:45:20.456887 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:20.457379 1237 external_mini_cluster.cc:1428] Setting key 5df53850997f03d832452fdd404843af
I20260927 22:45:20.468307 18995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0: Bootstrap starting.
I20260927 22:45:20.472460 18995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:20.473728 18995 log.cc:824] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:20.474469 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:20.459537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.1.53.126:37609)
0927 22:45:20.460741 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:20.460757 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:20.460811 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:45:20.463192 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:20.463221 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:20.463869 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:20.464372 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:20.464956 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:20.467042 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:20.468884 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:20.469909 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:20.470133 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:20.471494 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:20.471505 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:20.471525 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:20.472259 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:20.472390 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:20.472419 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:20.472554 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:20.472926 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:45:20.473280 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":392,"threads_started":1}
I20260927 22:45:20.476858 18995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0: No bootstrap required, opened a new log
I20260927 22:45:20.484725 18995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3588e69643bd47569b9b7d619a7e7be0" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37609 } }
I20260927 22:45:20.485137 18995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:20.485253 18995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3588e69643bd47569b9b7d619a7e7be0, State: Initialized, Role: FOLLOWER
I20260927 22:45:20.485878 18995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3588e69643bd47569b9b7d619a7e7be0" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37609 } }
I20260927 22:45:20.486148 18995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:20.486304 18995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:20.486471 18995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:20.488996 18995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3588e69643bd47569b9b7d619a7e7be0" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37609 } }
I20260927 22:45:20.489557 18995 leader_election.cc:302] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3588e69643bd47569b9b7d619a7e7be0; no voters:
I20260927 22:45:20.490221 18995 leader_election.cc:288] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:20.490377 19000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:20.491179 19000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [term 1 LEADER]: Becoming Leader. State: Replica: 3588e69643bd47569b9b7d619a7e7be0, State: Running, Role: LEADER
I20260927 22:45:20.491750 19000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3588e69643bd47569b9b7d619a7e7be0" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37609 } }
I20260927 22:45:20.492461 18995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:20.495226 19002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3588e69643bd47569b9b7d619a7e7be0. Latest consensus state: current_term: 1 leader_uuid: "3588e69643bd47569b9b7d619a7e7be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3588e69643bd47569b9b7d619a7e7be0" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37609 } } }
I20260927 22:45:20.495332 19001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3588e69643bd47569b9b7d619a7e7be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3588e69643bd47569b9b7d619a7e7be0" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37609 } } }
I20260927 22:45:20.495754 19002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:20.495777 19001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:20.501068 19008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:20.507234 19008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:20.516844 19008 catalog_manager.cc:1347] Generated new cluster ID: 29ca390c9ee14a72b34d07e0eb573217
I20260927 22:45:20.516992 19008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:20.530797 19008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:20.532387 19008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:20.556417 19008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0: Generated new TSK 0
I20260927 22:45:20.557631 19008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:20.571520 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37609
--builtin_ntp_servers=127.1.53.84:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:45:20.930856 19019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:20.931164 19019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:20.931313 19019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:20.938901 19019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:20.939011 19019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:20.939193 19019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:45:20.948972 19019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:20.950398 19019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:20.952230 19019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:20.965848 19025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:20.965869 19027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:20.966516 19024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:20.966542 19019 server_base.cc:1034] running on GCE node
I20260927 22:45:20.967284 19019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:20.968290 19019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:20.969515 19019 hybrid_clock.cc:648] HybridClock initialized: now 1790549120969474 us; error 33 us; skew 500 ppm
I20260927 22:45:20.969884 19019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:20.972138 19019 webserver.cc:460] Webserver started at http://127.1.53.65:46879/ using document root <none> and password file <none>
I20260927 22:45:20.972893 19019 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:20.972981 19019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:20.973443 19019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:20.977299 19019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "ece6df87bf8d418594f16e1e2d438808"
format_stamp: "Formatted at 2026-09-27 22:45:20 on dist-test-slave-pp80"
server_key: "a48b280b661c9399aa9beb2fb69a6672"
server_key_iv: "d422a11006868e4e382018da1d7cb959"
server_key_version: "encryptionkey@0"
I20260927 22:45:20.978039 19019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "ece6df87bf8d418594f16e1e2d438808"
format_stamp: "Formatted at 2026-09-27 22:45:20 on dist-test-slave-pp80"
server_key: "a48b280b661c9399aa9beb2fb69a6672"
server_key_iv: "d422a11006868e4e382018da1d7cb959"
server_key_version: "encryptionkey@0"
I20260927 22:45:20.983352 19019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:45:20.986907 19033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:20.988353 19019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:20.988526 19019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "ece6df87bf8d418594f16e1e2d438808"
format_stamp: "Formatted at 2026-09-27 22:45:20 on dist-test-slave-pp80"
server_key: "a48b280b661c9399aa9beb2fb69a6672"
server_key_iv: "d422a11006868e4e382018da1d7cb959"
server_key_version: "encryptionkey@0"
I20260927 22:45:20.988761 19019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:21.006225 19019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:21.007054 19019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:21.007294 19019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:21.008320 19019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:21.009872 19019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:21.009963 19019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:21.010044 19019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:21.010097 19019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:21.044029 19019 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:40165
I20260927 22:45:21.044116 19145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:40165 every 8 connection(s)
I20260927 22:45:21.045459 19019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:45:21.050580 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19019
I20260927 22:45:21.050755 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:45:21.055768 1237 external_mini_cluster.cc:1428] Setting key 8ea102214c36b9b380b1c1059cb04c58
I20260927 22:45:21.058790 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37609
--builtin_ntp_servers=127.1.53.84:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:45:21.060536 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:21.048083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:60515 (local address 127.1.53.126:37609)
0927 22:45:21.048555 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:21.048560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:21.049729 (+ 1169us) server_negotiation.cc:407] Connection header received
0927 22:45:21.052482 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:21.052486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:21.052555 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:21.052651 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:21.054228 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.055101 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.057227 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.057712 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.057828 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:21.059461 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:21.059470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:21.059478 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:21.059506 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:21.059572 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:21.059591 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:21.059691 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:21.059915 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:45:21.060044 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":269,"threads_started":1}
I20260927 22:45:21.061519 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:21.048641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37609 (local address 127.1.53.65:60515)
0927 22:45:21.049548 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:21.049582 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:21.052101 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:21.052979 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:21.052994 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:21.053134 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:21.054037 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:21.054062 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.055298 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.055304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:21.057082 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:21.057097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.057955 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.057965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:21.058230 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:21.059042 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:21.059046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:21.059113 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:21.059234 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:21.059697 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:21.059706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:21.059712 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:45:21.059863 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 22:45:21.060346 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":283,"threads_started":1}
I20260927 22:45:21.063174 19146 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37609
I20260927 22:45:21.063514 19146 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:21.064515 19146 heartbeater.cc:511] Master 127.1.53.126:37609 requested a full tablet report, sending...
I20260927 22:45:21.066933 18960 ts_manager.cc:194] Registered new tserver with Master: ece6df87bf8d418594f16e1e2d438808 (127.1.53.65:40165)
I20260927 22:45:21.069257 18960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.65:60515
W20260927 22:45:21.398020 19150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:21.398300 19150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:21.398394 19150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:21.406071 19150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:21.406188 19150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:21.406296 19150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:45:21.415602 19150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:21.416900 19150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:21.418620 19150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:21.432446 19158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:21.432545 19156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:21.432475 19155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:21.438405 19150 server_base.cc:1034] running on GCE node
I20260927 22:45:21.439149 19150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:21.440320 19150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:21.441529 19150 hybrid_clock.cc:648] HybridClock initialized: now 1790549121441474 us; error 37 us; skew 500 ppm
I20260927 22:45:21.442025 19150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:21.444625 19150 webserver.cc:460] Webserver started at http://127.1.53.66:33603/ using document root <none> and password file <none>
I20260927 22:45:21.445396 19150 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:21.445488 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:21.445823 19150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:21.449061 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "52f632b1d15641e9a6607db401949910"
format_stamp: "Formatted at 2026-09-27 22:45:21 on dist-test-slave-pp80"
server_key: "3667f36fb78407916e66da0d69706eb3"
server_key_iv: "fc1f2f35401c69f0c99f43318027ba6e"
server_key_version: "encryptionkey@0"
I20260927 22:45:21.449888 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "52f632b1d15641e9a6607db401949910"
format_stamp: "Formatted at 2026-09-27 22:45:21 on dist-test-slave-pp80"
server_key: "3667f36fb78407916e66da0d69706eb3"
server_key_iv: "fc1f2f35401c69f0c99f43318027ba6e"
server_key_version: "encryptionkey@0"
I20260927 22:45:21.455358 19150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:45:21.459108 19164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:21.460840 19150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:21.461036 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "52f632b1d15641e9a6607db401949910"
format_stamp: "Formatted at 2026-09-27 22:45:21 on dist-test-slave-pp80"
server_key: "3667f36fb78407916e66da0d69706eb3"
server_key_iv: "fc1f2f35401c69f0c99f43318027ba6e"
server_key_version: "encryptionkey@0"
I20260927 22:45:21.461292 19150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:21.487413 19150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:21.488329 19150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:21.488634 19150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:21.489711 19150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:21.491232 19150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:21.491361 19150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:21.491500 19150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:21.491612 19150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:21.526892 19150 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:40429
I20260927 22:45:21.526950 19276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:40429 every 8 connection(s)
I20260927 22:45:21.528360 19150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:45:21.537748 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19150
I20260927 22:45:21.538008 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:45:21.538661 1237 external_mini_cluster.cc:1428] Setting key 1c4dd9459dae2dbb444cf027435a4499
I20260927 22:45:21.542641 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37609
--builtin_ntp_servers=127.1.53.84:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:45:21.543910 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:21.531228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:58131 (local address 127.1.53.126:37609)
0927 22:45:21.531441 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:21.531446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:21.533159 (+ 1713us) server_negotiation.cc:407] Connection header received
0927 22:45:21.535165 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:21.535169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:21.535241 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:21.535337 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:21.536734 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.537496 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.540260 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.541189 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.541326 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:21.542997 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:21.543003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:21.543008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:21.543035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:21.543066 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:21.543078 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:21.543231 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:21.543476 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:45:21.543606 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:45:21.545256 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:21.531721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37609 (local address 127.1.53.66:58131)
0927 22:45:21.532887 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:21.532999 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:21.534848 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:21.535590 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:21.535599 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:21.535709 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:21.536551 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:21.536572 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.537802 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.537807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:21.540001 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:21.540018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:21.541451 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:21.541460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:21.541749 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:21.542551 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:21.542553 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:21.542598 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:21.542698 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:21.543229 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:21.543242 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:21.543252 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:45:21.543468 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 22:45:21.543968 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":361,"threads_started":1}
I20260927 22:45:21.546896 19277 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37609
I20260927 22:45:21.547245 19277 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:21.548008 19277 heartbeater.cc:511] Master 127.1.53.126:37609 requested a full tablet report, sending...
I20260927 22:45:21.549567 18960 ts_manager.cc:194] Registered new tserver with Master: 52f632b1d15641e9a6607db401949910 (127.1.53.66:40429)
I20260927 22:45:21.550423 18960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.66:58131
W20260927 22:45:21.907002 19280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:21.907331 19280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:21.907426 19280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:21.915405 19280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:21.915524 19280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:21.915655 19280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:45:21.924965 19280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:21.926307 19280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:21.928018 19280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:21.941440 19285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:21.941409 19288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:21.941376 19286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:21.942255 19280 server_base.cc:1034] running on GCE node
I20260927 22:45:21.942869 19280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:21.944080 19280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:21.945307 19280 hybrid_clock.cc:648] HybridClock initialized: now 1790549121945235 us; error 58 us; skew 500 ppm
I20260927 22:45:21.945659 19280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:21.948091 19280 webserver.cc:460] Webserver started at http://127.1.53.67:46441/ using document root <none> and password file <none>
I20260927 22:45:21.948843 19280 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:21.948964 19280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:21.949322 19280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:21.952190 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "82752658e36943ddbca9f7684d05c65e"
format_stamp: "Formatted at 2026-09-27 22:45:21 on dist-test-slave-pp80"
server_key: "3abfe559d724c8260ee2a3b2b0b6bbd3"
server_key_iv: "d4b459675733b0e89c5a14df8ef1d41c"
server_key_version: "encryptionkey@0"
I20260927 22:45:21.953071 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "82752658e36943ddbca9f7684d05c65e"
format_stamp: "Formatted at 2026-09-27 22:45:21 on dist-test-slave-pp80"
server_key: "3abfe559d724c8260ee2a3b2b0b6bbd3"
server_key_iv: "d4b459675733b0e89c5a14df8ef1d41c"
server_key_version: "encryptionkey@0"
I20260927 22:45:21.958195 19280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:45:21.961877 19294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:21.963320 19280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:45:21.963510 19280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "82752658e36943ddbca9f7684d05c65e"
format_stamp: "Formatted at 2026-09-27 22:45:21 on dist-test-slave-pp80"
server_key: "3abfe559d724c8260ee2a3b2b0b6bbd3"
server_key_iv: "d4b459675733b0e89c5a14df8ef1d41c"
server_key_version: "encryptionkey@0"
I20260927 22:45:21.963765 19280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:21.997746 19280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:21.998677 19280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:21.998975 19280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:21.999981 19280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:22.001776 19280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:22.001896 19280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:22.002028 19280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:22.002128 19280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:22.036298 19280 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:45037
I20260927 22:45:22.036365 19406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:45037 every 8 connection(s)
I20260927 22:45:22.037663 19280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:45:22.041208 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19280
I20260927 22:45:22.041414 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:45:22.041980 1237 external_mini_cluster.cc:1428] Setting key 1095cf73fd0ee20c24c889989a9c91f9
I20260927 22:45:22.052085 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.040402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:53725 (local address 127.1.53.126:37609)
0927 22:45:22.040584 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.040588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.042293 (+ 1705us) server_negotiation.cc:407] Connection header received
0927 22:45:22.044674 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.044691 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.044797 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.044900 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:22.046502 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.047263 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.049078 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.049521 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.049677 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.051218 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:22.051224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:22.051227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:22.051247 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:22.051281 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:22.051293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:22.051399 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.051645 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.051754 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:45:22.053278 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.040909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37609 (local address 127.1.53.67:53725)
0927 22:45:22.042054 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:22.042104 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:22.044321 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:22.045210 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:22.045249 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:22.045367 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:22.046304 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.046328 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.047440 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.047451 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.048903 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.048917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.049785 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.049793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.050042 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.050767 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:22.050774 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:22.050816 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:22.050906 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:22.051400 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:22.051414 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:22.051424 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:45:22.051635 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 22:45:22.052090 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260927 22:45:22.054805 19407 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37609
I20260927 22:45:22.055154 19407 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:22.055953 19407 heartbeater.cc:511] Master 127.1.53.126:37609 requested a full tablet report, sending...
I20260927 22:45:22.057797 18959 ts_manager.cc:194] Registered new tserver with Master: 82752658e36943ddbca9f7684d05c65e (127.1.53.67:45037)
I20260927 22:45:22.058943 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:45:22.059000 18959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.67:53725
I20260927 22:45:22.070307 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.063843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.1.53.126:37609)
0927 22:45:22.064061 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.064070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.064528 (+ 458us) server_negotiation.cc:407] Connection header received
0927 22:45:22.064768 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.064774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.064875 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.064988 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:22.065517 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.066233 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.067703 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.068221 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.068390 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.069599 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:22.069605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:22.069608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:22.069637 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:22.069671 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:22.069683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:22.069764 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.069960 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.070048 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:45:22.072620 19146 heartbeater.cc:503] Master 127.1.53.126:37609 was elected leader, sending a full tablet report...
I20260927 22:45:22.075214 18959 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:45:22.082026 18959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:45:22.117995 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.095025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:45037 (local address 127.0.0.1:43194)
0927 22:45:22.106440 (+ 11415us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:22.106481 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:22.106681 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:22.109512 (+ 2831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:22.109521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:22.109556 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:22.110093 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.110114 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.112087 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.112091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.114848 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.114888 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.116898 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.116907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.117210 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.117271 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 22:45:22.117403 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:45:22.117587 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11135,"mutex_wait_us":31,"thread_start_us":17243,"threads_started":1}
I20260927 22:45:22.119059 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.095679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.1.53.67:45037)
0927 22:45:22.108662 (+ 12983us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.108685 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.108732 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:45:22.108864 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.108876 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.109226 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.109502 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:22.110371 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.111963 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.115122 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.116693 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.116834 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.117134 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.118485 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.118685 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12726,"thread_start_us":12534,"threads_started":1}
I20260927 22:45:22.119609 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.095103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:40429 (local address 127.0.0.1:34578)
0927 22:45:22.110256 (+ 15153us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:22.110276 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:22.110423 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:22.112053 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:22.112057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:22.112087 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:22.112679 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.112692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.114593 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.114598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.117172 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.117190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.118691 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.118698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.118854 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.118886 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:45:22.118970 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:45:22.119111 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15010,"mutex_wait_us":144,"thread_start_us":14855,"threads_started":1}
I20260927 22:45:22.120030 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.097529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.1.53.65:40165)
0927 22:45:22.105755 (+ 8226us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.105771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.112151 (+ 6380us) server_negotiation.cc:407] Connection header received
0927 22:45:22.112312 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.112320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.112559 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.112763 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:22.113369 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.114473 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.117092 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.118570 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.118689 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.118957 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.119481 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.119672 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8014,"thread_start_us":7868,"threads_started":1}
I20260927 22:45:22.120028 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.096843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:40165 (local address 127.0.0.1:58156)
0927 22:45:22.110852 (+ 14009us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:22.110872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:22.110981 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:22.112754 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:22.112758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:22.112785 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:22.113190 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.113199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.114616 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.114631 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.116852 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.116868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.119091 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.119098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.119282 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.119313 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:45:22.119436 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:45:22.119565 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13891,"mutex_wait_us":95,"thread_start_us":9136,"threads_started":1}
I20260927 22:45:22.122864 19342 tablet_service.cc:1467] Processing CreateTablet for tablet 57544e4cca864184a740497170107792 (DEFAULT_TABLE table=test-table [id=d3615e585689491d9f994161b54c57cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:22.124179 19342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57544e4cca864184a740497170107792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:22.124827 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.095225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.1.53.66:40429)
0927 22:45:22.111438 (+ 16213us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.111447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.111476 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:45:22.111565 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.111573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.111827 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.112024 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:22.112899 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.114440 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.117384 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.118485 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.118651 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.118876 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.119023 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.119185 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16054,"thread_start_us":15938,"threads_started":1}
I20260927 22:45:22.126111 19081 tablet_service.cc:1467] Processing CreateTablet for tablet 57544e4cca864184a740497170107792 (DEFAULT_TABLE table=test-table [id=d3615e585689491d9f994161b54c57cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:22.127651 19081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57544e4cca864184a740497170107792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:22.128768 19212 tablet_service.cc:1467] Processing CreateTablet for tablet 57544e4cca864184a740497170107792 (DEFAULT_TABLE table=test-table [id=d3615e585689491d9f994161b54c57cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:22.130194 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57544e4cca864184a740497170107792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:22.139068 19425 tablet_bootstrap.cc:492] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Bootstrap starting.
I20260927 22:45:22.141363 19426 tablet_bootstrap.cc:492] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Bootstrap starting.
I20260927 22:45:22.143117 19425 tablet_bootstrap.cc:654] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:22.144797 19425 log.cc:824] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:22.145123 19426 tablet_bootstrap.cc:654] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:22.146487 19426 log.cc:824] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:22.147033 19427 tablet_bootstrap.cc:492] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Bootstrap starting.
I20260927 22:45:22.148089 19425 tablet_bootstrap.cc:492] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: No bootstrap required, opened a new log
I20260927 22:45:22.148425 19425 ts_tablet_manager.cc:1397] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 22:45:22.149711 19426 tablet_bootstrap.cc:492] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: No bootstrap required, opened a new log
I20260927 22:45:22.149969 19426 ts_tablet_manager.cc:1397] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 22:45:22.150353 19427 tablet_bootstrap.cc:654] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:22.151314 19427 log.cc:824] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:22.153990 19427 tablet_bootstrap.cc:492] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: No bootstrap required, opened a new log
I20260927 22:45:22.154287 19427 ts_tablet_manager.cc:1397] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:45:22.159292 19425 raft_consensus.cc:348] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.159789 19425 raft_consensus.cc:374] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:22.159899 19425 raft_consensus.cc:729] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece6df87bf8d418594f16e1e2d438808, State: Initialized, Role: FOLLOWER
I20260927 22:45:22.160463 19426 raft_consensus.cc:348] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.160637 19425 consensus_queue.cc:260] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.160951 19426 raft_consensus.cc:374] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:22.161057 19426 raft_consensus.cc:729] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82752658e36943ddbca9f7684d05c65e, State: Initialized, Role: FOLLOWER
I20260927 22:45:22.161800 19426 consensus_queue.cc:260] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.162570 19425 ts_tablet_manager.cc:1428] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 22:45:22.162613 19427 raft_consensus.cc:348] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.163017 19427 raft_consensus.cc:374] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:22.163137 19427 raft_consensus.cc:729] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f632b1d15641e9a6607db401949910, State: Initialized, Role: FOLLOWER
I20260927 22:45:22.163908 19407 heartbeater.cc:503] Master 127.1.53.126:37609 was elected leader, sending a full tablet report...
I20260927 22:45:22.163811 19427 consensus_queue.cc:260] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.164613 19426 ts_tablet_manager.cc:1428] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 22:45:22.166563 19277 heartbeater.cc:503] Master 127.1.53.126:37609 was elected leader, sending a full tablet report...
I20260927 22:45:22.167817 19427 ts_tablet_manager.cc:1428] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260927 22:45:22.180053 19431 raft_consensus.cc:482] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:22.180442 19431 raft_consensus.cc:504] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.182147 19431 leader_election.cc:288] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f632b1d15641e9a6607db401949910 (127.1.53.66:40429), 82752658e36943ddbca9f7684d05c65e (127.1.53.67:45037)
I20260927 22:45:22.189915 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.182596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:41331 (local address 127.1.53.67:45037)
0927 22:45:22.182804 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.182811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.183629 (+ 818us) server_negotiation.cc:407] Connection header received
0927 22:45:22.183808 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.183814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.183896 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.183994 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:22.185048 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.186045 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.188135 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.188997 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.189134 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.189257 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.189544 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.189670 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:45:22.189917 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.182528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:34155 (local address 127.1.53.66:40429)
0927 22:45:22.182767 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.182776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.183497 (+ 721us) server_negotiation.cc:407] Connection header received
0927 22:45:22.183821 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.183827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.183912 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.184087 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:22.185019 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.185905 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.188094 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.188919 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.189018 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.189125 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.189524 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.189659 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:45:22.190076 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.182581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:45037 (local address 127.1.53.65:41331)
0927 22:45:22.183504 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:22.183523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:22.183658 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:22.184026 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:22.184030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:22.184081 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:22.184755 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.184782 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.186199 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.186203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.187914 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.187926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.189206 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.189211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.189341 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.189386 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:45:22.189500 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:45:22.189629 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":186,"spinlock_wait_cycles":18816,"thread_start_us":514,"threads_started":1}
I20260927 22:45:22.190284 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.182393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:40429 (local address 127.1.53.65:34155)
0927 22:45:22.183350 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:22.183366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:22.183558 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:22.184104 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:22.184111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:22.184132 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:22.184855 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.184867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.186062 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.186067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.187889 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:22.187905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.189114 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.189121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:22.189299 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.189372 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 22:45:22.189447 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:45:22.189614 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":164,"spinlock_wait_cycles":40192,"thread_start_us":599,"threads_started":1}
I20260927 22:45:22.191263 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57544e4cca864184a740497170107792" candidate_uuid: "ece6df87bf8d418594f16e1e2d438808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82752658e36943ddbca9f7684d05c65e" is_pre_election: true
I20260927 22:45:22.191272 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57544e4cca864184a740497170107792" candidate_uuid: "ece6df87bf8d418594f16e1e2d438808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f632b1d15641e9a6607db401949910" is_pre_election: true
I20260927 22:45:22.191876 19232 raft_consensus.cc:2454] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece6df87bf8d418594f16e1e2d438808 in term 0.
I20260927 22:45:22.191897 19362 raft_consensus.cc:2454] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece6df87bf8d418594f16e1e2d438808 in term 0.
I20260927 22:45:22.192632 19037 leader_election.cc:302] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82752658e36943ddbca9f7684d05c65e, ece6df87bf8d418594f16e1e2d438808; no voters:
I20260927 22:45:22.193153 19431 raft_consensus.cc:2781] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:45:22.193343 19431 raft_consensus.cc:482] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:45:22.193434 19431 raft_consensus.cc:3037] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:22.195924 19431 raft_consensus.cc:504] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.196666 19431 leader_election.cc:288] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [CANDIDATE]: Term 1 election: Requested vote from peers 52f632b1d15641e9a6607db401949910 (127.1.53.66:40429), 82752658e36943ddbca9f7684d05c65e (127.1.53.67:45037)
I20260927 22:45:22.197177 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57544e4cca864184a740497170107792" candidate_uuid: "ece6df87bf8d418594f16e1e2d438808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f632b1d15641e9a6607db401949910"
I20260927 22:45:22.197270 19362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57544e4cca864184a740497170107792" candidate_uuid: "ece6df87bf8d418594f16e1e2d438808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82752658e36943ddbca9f7684d05c65e"
I20260927 22:45:22.197402 19232 raft_consensus.cc:3037] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:22.197517 19362 raft_consensus.cc:3037] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:22.199730 19232 raft_consensus.cc:2454] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece6df87bf8d418594f16e1e2d438808 in term 1.
I20260927 22:45:22.199748 19362 raft_consensus.cc:2454] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece6df87bf8d418594f16e1e2d438808 in term 1.
I20260927 22:45:22.200212 19035 leader_election.cc:302] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f632b1d15641e9a6607db401949910, ece6df87bf8d418594f16e1e2d438808; no voters:
I20260927 22:45:22.200639 19431 raft_consensus.cc:2781] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:22.201160 19431 raft_consensus.cc:686] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 1 LEADER]: Becoming Leader. State: Replica: ece6df87bf8d418594f16e1e2d438808, State: Running, Role: LEADER
I20260927 22:45:22.201658 19431 consensus_queue.cc:237] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } }
I20260927 22:45:22.206411 18960 catalog_manager.cc:5495] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 reported cstate change: term changed from 0 to 1, leader changed from <none> to ece6df87bf8d418594f16e1e2d438808 (127.1.53.65). New cstate: current_term: 1 leader_uuid: "ece6df87bf8d418594f16e1e2d438808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece6df87bf8d418594f16e1e2d438808" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40165 } health_report { overall_health: HEALTHY } } }
I20260927 22:45:22.241940 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.236181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.1.53.65:40165)
0927 22:45:22.236405 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.236412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.236456 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:45:22.236723 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.236727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.236821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.236986 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:45:22.237348 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.238049 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.239645 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.240065 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.240223 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.240831 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:45:22.241361 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:45:22.241537 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.241587 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.241672 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:45:22.250645 19232 raft_consensus.cc:1261] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 1 FOLLOWER]: Refusing update from remote peer ece6df87bf8d418594f16e1e2d438808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:22.250664 19362 raft_consensus.cc:1261] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 1 FOLLOWER]: Refusing update from remote peer ece6df87bf8d418594f16e1e2d438808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:22.251719 19436 consensus_queue.cc:1035] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f632b1d15641e9a6607db401949910" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:22.252084 19431 consensus_queue.cc:1035] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82752658e36943ddbca9f7684d05c65e" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 45037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:22.270598 19439 mvcc.cc:205] Tried to move back new op lower bound from 7334089204728389632 to 7334089204542230528. Current Snapshot: MvccSnapshot[applied={T|T < 7334089204728389632}]
I20260927 22:45:22.272781 19440 mvcc.cc:205] Tried to move back new op lower bound from 7334089204728389632 to 7334089204542230528. Current Snapshot: MvccSnapshot[applied={T|T < 7334089204728389632}]
I20260927 22:45:22.276147 19441 mvcc.cc:205] Tried to move back new op lower bound from 7334089204728389632 to 7334089204542230528. Current Snapshot: MvccSnapshot[applied={T|T < 7334089204728389632}]
I20260927 22:45:22.289919 18958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:22.290270 18958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:22.294704 18958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3588e69643bd47569b9b7d619a7e7be0: Sending DeleteTablet for 3 replicas of tablet 57544e4cca864184a740497170107792
I20260927 22:45:22.296025 19342 tablet_service.cc:1514] Processing DeleteTablet for tablet 57544e4cca864184a740497170107792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:22 UTC) from {username='slave'} at 127.0.0.1:43194
I20260927 22:45:22.296057 19212 tablet_service.cc:1514] Processing DeleteTablet for tablet 57544e4cca864184a740497170107792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:22 UTC) from {username='slave'} at 127.0.0.1:34578
I20260927 22:45:22.296797 19081 tablet_service.cc:1514] Processing DeleteTablet for tablet 57544e4cca864184a740497170107792 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:22 UTC) from {username='slave'} at 127.0.0.1:58156
I20260927 22:45:22.297065 19449 tablet_replica.cc:330] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: stopping tablet replica
I20260927 22:45:22.297560 19449 raft_consensus.cc:2229] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:22.297778 19450 tablet_replica.cc:330] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: stopping tablet replica
I20260927 22:45:22.298704 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19019
I20260927 22:45:22.298869 19449 raft_consensus.cc:2258] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:22.299767 19451 tablet_replica.cc:330] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: stopping tablet replica
I20260927 22:45:22.300272 19451 raft_consensus.cc:2229] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:22.300773 19451 raft_consensus.cc:2258] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:22.301081 19449 ts_tablet_manager.cc:1905] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:22.303216 19451 ts_tablet_manager.cc:1905] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:22.308683 19449 ts_tablet_manager.cc:1918] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:22.308874 19449 log.cc:1196] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/57544e4cca864184a740497170107792
I20260927 22:45:22.308161 19450 raft_consensus.cc:2229] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:45:22.309412 19449 ts_tablet_manager.cc:1939] T 57544e4cca864184a740497170107792 P 82752658e36943ddbca9f7684d05c65e: Deleting consensus metadata
I20260927 22:45:22.310212 19450 raft_consensus.cc:2258] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:22.311391 18947 catalog_manager.cc:4841] TS 82752658e36943ddbca9f7684d05c65e (127.1.53.67:45037): tablet 57544e4cca864184a740497170107792 (table test-table [id=d3615e585689491d9f994161b54c57cd]) successfully deleted
I20260927 22:45:22.311866 19451 ts_tablet_manager.cc:1918] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:22.312033 19451 log.cc:1196] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/57544e4cca864184a740497170107792
I20260927 22:45:22.312645 19451 ts_tablet_manager.cc:1939] T 57544e4cca864184a740497170107792 P 52f632b1d15641e9a6607db401949910: Deleting consensus metadata
I20260927 22:45:22.313454 19450 ts_tablet_manager.cc:1905] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:22.314477 18945 catalog_manager.cc:4841] TS 52f632b1d15641e9a6607db401949910 (127.1.53.66:40429): tablet 57544e4cca864184a740497170107792 (table test-table [id=d3615e585689491d9f994161b54c57cd]) successfully deleted
I20260927 22:45:22.320601 19450 ts_tablet_manager.cc:1918] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:22.320791 19450 log.cc:1196] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/57544e4cca864184a740497170107792
I20260927 22:45:22.321357 19450 ts_tablet_manager.cc:1939] T 57544e4cca864184a740497170107792 P ece6df87bf8d418594f16e1e2d438808: Deleting consensus metadata
I20260927 22:45:22.322872 18944 catalog_manager.cc:4841] TS ece6df87bf8d418594f16e1e2d438808 (127.1.53.65:40165): tablet 57544e4cca864184a740497170107792 (table test-table [id=d3615e585689491d9f994161b54c57cd]) successfully deleted
I20260927 22:45:22.334828 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.300197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.1.53.65:40165)
0927 22:45:22.300386 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.300394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.315501 (+ 15107us) server_negotiation.cc:407] Connection header received
0927 22:45:22.315707 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.315712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.315829 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.315949 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:22.316470 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.317767 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.319747 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.320444 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.320589 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.332920 (+ 12331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:22.332934 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:22.332955 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:22.333845 (+ 890us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:22.333918 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:22.333948 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:22.334094 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.334299 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.334439 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:45:22.335807 19141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:22.445950 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19019
I20260927 22:45:22.462651 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19150
I20260927 22:45:22.469815 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.463522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.1.53.66:40429)
0927 22:45:22.463709 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.463714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.463737 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:45:22.463912 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.463916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.464001 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.464116 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:22.464597 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.465515 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.466835 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.467227 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.467374 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.468378 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:22.468391 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:22.468413 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:22.469174 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:22.469220 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:22.469240 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:22.469327 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.469469 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.469554 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:45:22.470459 19272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:22.561183 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19150
I20260927 22:45:22.578640 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19280
I20260927 22:45:22.585656 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:22.579544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.1.53.67:45037)
0927 22:45:22.579732 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:22.579736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:22.579759 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:45:22.579940 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:22.579945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:22.580042 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:22.580143 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:22.580576 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.581313 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.582668 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:22.583038 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:22.583137 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:22.584220 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:22.584229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:22.584299 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:22.584871 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:22.584922 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:22.584941 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:22.585026 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:22.585213 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 22:45:22.585303 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:45:22.586470 19402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:22.676805 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19280
I20260927 22:45:22.693419 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 18929
I20260927 22:45:22.694581 18990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:22.806715 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 18929
2026-09-27T22:45:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2902 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:22 dist-test-slave-pp80 krb5kdc[19459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:22 dist-test-slave-pp80 krb5kdc[19459](info): set up 2 sockets
Sep 27 22:45:22 dist-test-slave-pp80 krb5kdc[19459](info): commencing operation
krb5kdc: starting...
W20260927 22:45:23.928555 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:24 dist-test-slave-pp80 krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:45:24.195550 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37791
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:24.543794 19472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:24.544109 19472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:24.544301 19472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:24.553437 19472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:24.553560 19472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:24.553640 19472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:24.553723 19472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:24.553812 19472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:24.564080 19472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:24.565541 19472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:24.566936 19472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:24.579825 19477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:24.579829 19478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:24.579733 19480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:24.581386 19472 server_base.cc:1034] running on GCE node
I20260927 22:45:24.582327 19472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:24.583932 19472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:24.585237 19472 hybrid_clock.cc:648] HybridClock initialized: now 1790549124585170 us; error 88 us; skew 500 ppm
I20260927 22:45:24.585664 19472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:24 dist-test-slave-pp80 krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:24.589092 19472 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:45:24.590906 19472 webserver.cc:460] Webserver started at http://127.1.53.126:45723/ using document root <none> and password file <none>
I20260927 22:45:24.591657 19472 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:24.591815 19472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:24.592145 19472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:24.595287 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "8584de1d3e0e4dce9dbd0b7022a18913"
format_stamp: "Formatted at 2026-09-27 22:45:24 on dist-test-slave-pp80"
server_key: "be33f13cc9949b1fdf6ea2ee843d2ba7"
server_key_iv: "65ef70d12a6b570a47e31e919a1e1f1c"
server_key_version: "encryptionkey@0"
I20260927 22:45:24.596406 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "8584de1d3e0e4dce9dbd0b7022a18913"
format_stamp: "Formatted at 2026-09-27 22:45:24 on dist-test-slave-pp80"
server_key: "be33f13cc9949b1fdf6ea2ee843d2ba7"
server_key_iv: "65ef70d12a6b570a47e31e919a1e1f1c"
server_key_version: "encryptionkey@0"
I20260927 22:45:24.602382 19472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:45:24.606232 19487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:24.607649 19472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:45:24.607853 19472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "8584de1d3e0e4dce9dbd0b7022a18913"
format_stamp: "Formatted at 2026-09-27 22:45:24 on dist-test-slave-pp80"
server_key: "be33f13cc9949b1fdf6ea2ee843d2ba7"
server_key_iv: "65ef70d12a6b570a47e31e919a1e1f1c"
server_key_version: "encryptionkey@0"
I20260927 22:45:24.608088 19472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:24.632668 19472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:24.637298 19472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:24.637634 19472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:24.658123 19472 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37791
I20260927 22:45:24.658161 19538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37791 every 8 connection(s)
I20260927 22:45:24.659797 19472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:24.661569 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19472
I20260927 22:45:24.661772 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:24.662376 1237 external_mini_cluster.cc:1428] Setting key 9419db16e3beb135f54488c4ae17018d
I20260927 22:45:24.664853 19539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:45:24 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:24.679229 19539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913: Bootstrap starting.
I20260927 22:45:24.683846 19539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:24.685248 19539 log.cc:824] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:24.689050 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:24.664464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.1.53.126:37791)
0927 22:45:24.665294 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:24.665306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:24.665350 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:45:24.667087 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:24.667111 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:24.667543 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:24.667898 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:24.669757 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:24.671187 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:24.673129 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:24.673940 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:24.674094 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:24.679570 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:24.679584 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:24.679612 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:24.679665 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:24.683876 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:24.684760 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:24.684769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:24.684779 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:24.685043 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:24.685422 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:24.685427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:24.685430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:24.686216 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:24.686450 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:24.687169 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 22:45:24.687649 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":399,"threads_started":1}
I20260927 22:45:24.689121 19539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913: No bootstrap required, opened a new log
I20260927 22:45:24.697466 19539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37791 } }
I20260927 22:45:24.697854 19539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:24.697921 19539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8584de1d3e0e4dce9dbd0b7022a18913, State: Initialized, Role: FOLLOWER
I20260927 22:45:24.698509 19539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37791 } }
I20260927 22:45:24.698711 19539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:24.698827 19539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:24.698951 19539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:24.701192 19539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37791 } }
I20260927 22:45:24.701650 19539 leader_election.cc:302] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8584de1d3e0e4dce9dbd0b7022a18913; no voters:
I20260927 22:45:24.702296 19539 leader_election.cc:288] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:24.702415 19544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:24.703771 19544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [term 1 LEADER]: Becoming Leader. State: Replica: 8584de1d3e0e4dce9dbd0b7022a18913, State: Running, Role: LEADER
I20260927 22:45:24.704406 19544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37791 } }
I20260927 22:45:24.704948 19539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:24.707846 19545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37791 } } }
I20260927 22:45:24.708093 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8584de1d3e0e4dce9dbd0b7022a18913. Latest consensus state: current_term: 1 leader_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8584de1d3e0e4dce9dbd0b7022a18913" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37791 } } }
I20260927 22:45:24.708348 19545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:24.708412 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:24.711753 19553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:24.716114 19553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:24.725878 19553 catalog_manager.cc:1347] Generated new cluster ID: e2dfbccef31b48f2aa5eede3037e1426
I20260927 22:45:24.726040 19553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:24.739428 19553 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:24.740952 19553 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:24.748478 19553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913: Generated new TSK 0
I20260927 22:45:24.749645 19553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:45:24.820981 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37791
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:25.169685 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:25.170007 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:25.170185 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:25.178294 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:25.178426 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:25.178615 19565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:45:25.189260 19565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:25.190698 19565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:25.192205 19565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:25.206869 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:25.206869 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:25.207379 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:25.207957 19565 server_base.cc:1034] running on GCE node
I20260927 22:45:25.208750 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:25.209895 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:25.211184 19565 hybrid_clock.cc:648] HybridClock initialized: now 1790549125211138 us; error 40 us; skew 500 ppm
I20260927 22:45:25.211562 19565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:25 dist-test-slave-pp80 krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:25.215019 19565 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:45:25.216810 19565 webserver.cc:460] Webserver started at http://127.1.53.65:36043/ using document root <none> and password file <none>
I20260927 22:45:25.217644 19565 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:25.217808 19565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:25.218148 19565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:25.221344 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "d162552dbd3244b58713269f85ace912"
format_stamp: "Formatted at 2026-09-27 22:45:25 on dist-test-slave-pp80"
server_key: "e1f57f5a6a8744495cbae6781607e952"
server_key_iv: "58b0070697b1955736ba561a61512896"
server_key_version: "encryptionkey@0"
I20260927 22:45:25.222187 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "d162552dbd3244b58713269f85ace912"
format_stamp: "Formatted at 2026-09-27 22:45:25 on dist-test-slave-pp80"
server_key: "e1f57f5a6a8744495cbae6781607e952"
server_key_iv: "58b0070697b1955736ba561a61512896"
server_key_version: "encryptionkey@0"
I20260927 22:45:25.227874 19565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:45:25.231817 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:25.233335 19565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:45:25.233559 19565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "d162552dbd3244b58713269f85ace912"
format_stamp: "Formatted at 2026-09-27 22:45:25 on dist-test-slave-pp80"
server_key: "e1f57f5a6a8744495cbae6781607e952"
server_key_iv: "58b0070697b1955736ba561a61512896"
server_key_version: "encryptionkey@0"
I20260927 22:45:25.233799 19565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:25.266892 19565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:25.271647 19565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:25.271975 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:25.273136 19565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:25.274609 19565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:25.274730 19565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:25.274902 19565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:25.274991 19565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:25.311514 19565 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:45859
I20260927 22:45:25.311635 19692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:45859 every 8 connection(s)
I20260927 22:45:25.313084 19565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:45:25.317549 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19565
I20260927 22:45:25.317760 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:45:25.318364 1237 external_mini_cluster.cc:1428] Setting key cbdf557040ad6e637690cc523c2dc378
Sep 27 22:45:25 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:25.338398 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:25.315909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:36995 (local address 127.1.53.126:37791)
0927 22:45:25.316488 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:25.316493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:25.317725 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 22:45:25.319983 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:25.319988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:25.320096 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:25.320297 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:25.322785 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.323507 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.326096 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.326511 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.326616 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:25.331503 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:25.331509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:25.331512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:25.331552 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:25.334342 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:25.335435 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:25.335441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:25.335444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:25.335610 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:25.336170 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:25.336173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:25.336175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:25.336539 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:25.336665 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:25.337817 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0927 22:45:25.338072 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":306,"threads_started":1}
I20260927 22:45:25.339905 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:25.316464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37791 (local address 127.1.53.65:36995)
0927 22:45:25.317476 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:25.317536 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:25.319695 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:25.320632 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:25.320646 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:25.321611 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:25.322565 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:25.322597 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.323694 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.323703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:25.325905 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:25.325955 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.326783 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.326794 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:25.327147 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:25.327927 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:25.327970 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:25.331208 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:25.334549 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:25.334563 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:25.334586 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:25.335282 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:25.335750 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:25.335758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:25.335764 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:25.336041 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:25.336674 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:25.336685 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:25.337230 (+ 545us) client_negotiation.cc:769] Sending connection context
0927 22:45:25.337618 (+ 388us) client_negotiation.cc:240] Negotiation successful
0927 22:45:25.338258 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":329,"threads_started":1}
I20260927 22:45:25.341753 19693 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37791
I20260927 22:45:25.342126 19693 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:25.343147 19693 heartbeater.cc:511] Master 127.1.53.126:37791 requested a full tablet report, sending...
I20260927 22:45:25.345707 19504 ts_manager.cc:194] Registered new tserver with Master: d162552dbd3244b58713269f85ace912 (127.1.53.65:45859)
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
I20260927 22:45:25.347360 19504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:36995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:45:25.384629 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37791
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:25.726680 19699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:25.727007 19699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:25.727106 19699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:25.734947 19699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:25.735083 19699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:25.735199 19699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:45:25.745677 19699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:25.747069 19699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:25.748570 19699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:25.762684 19704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:25.762712 19705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:25.762758 19707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:25.763648 19699 server_base.cc:1034] running on GCE node
I20260927 22:45:25.764488 19699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:25.765539 19699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:25.766753 19699 hybrid_clock.cc:648] HybridClock initialized: now 1790549125766709 us; error 32 us; skew 500 ppm
I20260927 22:45:25.767105 19699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:25 dist-test-slave-pp80 krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:25.770622 19699 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:45:25.772396 19699 webserver.cc:460] Webserver started at http://127.1.53.66:43303/ using document root <none> and password file <none>
I20260927 22:45:25.773155 19699 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:25.773309 19699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:25.773641 19699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:25.776778 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "05e6997c770644159fafb2835fa5fe07"
format_stamp: "Formatted at 2026-09-27 22:45:25 on dist-test-slave-pp80"
server_key: "ed5751728e08d9bf4fc511dc2362f16a"
server_key_iv: "0b3c5f2bcd37035518b37683c0db0466"
server_key_version: "encryptionkey@0"
I20260927 22:45:25.777616 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "05e6997c770644159fafb2835fa5fe07"
format_stamp: "Formatted at 2026-09-27 22:45:25 on dist-test-slave-pp80"
server_key: "ed5751728e08d9bf4fc511dc2362f16a"
server_key_iv: "0b3c5f2bcd37035518b37683c0db0466"
server_key_version: "encryptionkey@0"
I20260927 22:45:25.783268 19699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:45:25.787017 19714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:25.788527 19699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:25.788725 19699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "05e6997c770644159fafb2835fa5fe07"
format_stamp: "Formatted at 2026-09-27 22:45:25 on dist-test-slave-pp80"
server_key: "ed5751728e08d9bf4fc511dc2362f16a"
server_key_iv: "0b3c5f2bcd37035518b37683c0db0466"
server_key_version: "encryptionkey@0"
I20260927 22:45:25.788944 19699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:25.821283 19699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:25.826156 19699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:25.826484 19699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:25.827512 19699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:25.828922 19699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:25.829036 19699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:25.829195 19699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:25.829280 19699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:25.864480 19699 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:34493
I20260927 22:45:25.864526 19826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:34493 every 8 connection(s)
I20260927 22:45:25.865888 19699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:45:25.870469 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19699
I20260927 22:45:25.870671 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:45:25.871217 1237 external_mini_cluster.cc:1428] Setting key c77d7b58a422f39565ef3bf60948db40
Sep 27 22:45:25 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:25.890262 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:25.868956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:55205 (local address 127.1.53.126:37791)
0927 22:45:25.869450 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:25.869456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:25.871873 (+ 2417us) server_negotiation.cc:407] Connection header received
0927 22:45:25.873379 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:25.873385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:25.873462 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:25.873582 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:25.875277 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.876520 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.878202 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.879004 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.879161 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:25.883520 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:25.883525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:25.883528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:25.883565 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:25.886611 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:25.887688 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:25.887692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:25.887694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:25.887827 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:25.888383 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:25.888386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:25.888388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:25.888675 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:25.888790 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:25.889765 (+ 975us) server_negotiation.cc:299] Negotiation successful
0927 22:45:25.889966 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260927 22:45:25.891878 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:25.869658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37791 (local address 127.1.53.66:55205)
0927 22:45:25.870558 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:25.870614 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:25.873040 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:25.873842 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:25.873852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:25.874417 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:25.875022 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:25.875096 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.876691 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.876696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:25.877978 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:25.877991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:25.879244 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:25.879250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:25.879483 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:25.880354 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:25.880380 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:25.883235 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:25.886805 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:25.886820 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:25.886843 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:25.887555 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:25.887949 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:25.887957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:25.887964 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:25.888274 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:25.888823 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:25.888835 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:25.889348 (+ 513us) client_negotiation.cc:769] Sending connection context
0927 22:45:25.889684 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 22:45:25.890329 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":368,"threads_started":1}
I20260927 22:45:25.893677 19827 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37791
I20260927 22:45:25.894055 19827 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:25.894913 19827 heartbeater.cc:511] Master 127.1.53.126:37791 requested a full tablet report, sending...
I20260927 22:45:25.896575 19504 ts_manager.cc:194] Registered new tserver with Master: 05e6997c770644159fafb2835fa5fe07 (127.1.53.66:34493)
I20260927 22:45:25.897481 19504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:55205
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:45:25.935894 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37791
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:26.284161 19833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:26.284595 19833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:26.284780 19833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:26.292423 19833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:26.292594 19833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:26.292789 19833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:45:26.304095 19833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:37791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:26.305667 19833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:26.307165 19833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:26.321619 19839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:26.321714 19841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:26.321719 19838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:26.322752 19833 server_base.cc:1034] running on GCE node
I20260927 22:45:26.323386 19833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:26.324477 19833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:26.325688 19833 hybrid_clock.cc:648] HybridClock initialized: now 1790549126325641 us; error 36 us; skew 500 ppm
I20260927 22:45:26.326056 19833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:26 dist-test-slave-pp80 krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549126, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:26.329487 19833 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:45:26.331156 19833 webserver.cc:460] Webserver started at http://127.1.53.67:44731/ using document root <none> and password file <none>
I20260927 22:45:26.331900 19833 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:26.332056 19833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:26.332465 19833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:26.335533 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "478851c25e5e40d6b98bc102261c20a0"
format_stamp: "Formatted at 2026-09-27 22:45:26 on dist-test-slave-pp80"
server_key: "d4d9fc272d9f182d4664593883236075"
server_key_iv: "1680a036b0cc0924a7895f0f0fec8e0b"
server_key_version: "encryptionkey@0"
I20260927 22:45:26.336473 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "478851c25e5e40d6b98bc102261c20a0"
format_stamp: "Formatted at 2026-09-27 22:45:26 on dist-test-slave-pp80"
server_key: "d4d9fc272d9f182d4664593883236075"
server_key_iv: "1680a036b0cc0924a7895f0f0fec8e0b"
server_key_version: "encryptionkey@0"
I20260927 22:45:26.342090 19833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:45:26.346060 19848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:26.347646 19833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:45:26.347872 19833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "478851c25e5e40d6b98bc102261c20a0"
format_stamp: "Formatted at 2026-09-27 22:45:26 on dist-test-slave-pp80"
server_key: "d4d9fc272d9f182d4664593883236075"
server_key_iv: "1680a036b0cc0924a7895f0f0fec8e0b"
server_key_version: "encryptionkey@0"
I20260927 22:45:26.348126 19833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:26.351300 19693 heartbeater.cc:503] Master 127.1.53.126:37791 was elected leader, sending a full tablet report...
I20260927 22:45:26.391703 19833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:26.396641 19833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:26.396957 19833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:26.397992 19833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:26.399369 19833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:26.399480 19833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:26.399632 19833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:26.399709 19833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:26.436373 19833 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:35029
I20260927 22:45:26.436437 19960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:35029 every 8 connection(s)
I20260927 22:45:26.437852 19833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:45:26.446316 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 19833
I20260927 22:45:26.446547 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:45:26.447176 1237 external_mini_cluster.cc:1428] Setting key fef3d60d07b532076c4e7312a9094a5f
Sep 27 22:45:26 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549126, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:26.461644 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.440942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:38175 (local address 127.1.53.126:37791)
0927 22:45:26.441444 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.441451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.442517 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 22:45:26.444440 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.444445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.444530 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.444633 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:26.446405 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.447601 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.450117 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.450793 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.450950 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.455058 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:26.455066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:26.455071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:26.455127 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:26.458249 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:26.459043 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:26.459048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:26.459050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:26.459178 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:26.459584 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:26.459587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:26.459589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:26.459871 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:26.459996 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.460944 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.461236 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":225,"threads_started":1}
I20260927 22:45:26.462944 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.441574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:37791 (local address 127.1.53.67:38175)
0927 22:45:26.442334 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:26.442378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:26.444094 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:26.444898 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:26.444908 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:26.445521 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:26.446180 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.446199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.447820 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.447827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.449926 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.449945 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.451079 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.451087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.451360 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.452167 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:26.452192 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:26.454802 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:26.458389 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:26.458396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:26.458409 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:26.458846 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:26.459284 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:26.459289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:26.459292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:26.459474 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:26.460076 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:26.460082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:26.460541 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 22:45:26.460846 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 22:45:26.461376 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":345,"threads_started":1}
I20260927 22:45:26.464448 19961 heartbeater.cc:347] Connected to a master server at 127.1.53.126:37791
I20260927 22:45:26.464804 19961 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:26.465570 19961 heartbeater.cc:511] Master 127.1.53.126:37791 requested a full tablet report, sending...
I20260927 22:45:26.467208 19504 ts_manager.cc:194] Registered new tserver with Master: 478851c25e5e40d6b98bc102261c20a0 (127.1.53.67:35029)
I20260927 22:45:26.467995 19504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:38175
I20260927 22:45:26.476575 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:45:26.494331 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.480764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.1.53.126:37791)
0927 22:45:26.480939 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.480944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.481369 (+ 425us) server_negotiation.cc:407] Connection header received
0927 22:45:26.481507 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.481511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.481575 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.481664 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:26.483060 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.483792 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.485271 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.485658 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.485754 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.488576 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:26.488581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:26.488584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:26.488621 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:26.491177 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:26.492008 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:26.492022 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:26.492027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:26.492162 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:26.492738 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:26.492741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:26.492743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:26.493127 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:26.493420 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.493838 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.494045 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:45:26.498566 19504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:45:26.506016 19504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:45:26.531658 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.519405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45859 (local address 127.0.0.1:41896)
0927 22:45:26.522244 (+ 2839us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:26.522263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:26.522470 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:26.523473 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:26.523484 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:26.523517 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:26.523982 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.523994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.526042 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.526048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.529063 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.529079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.530565 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.530572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.530764 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.530822 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:45:26.530960 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:45:26.531175 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2712,"mutex_wait_us":87,"thread_start_us":392,"threads_started":1}
I20260927 22:45:26.532471 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.519178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:35029 (local address 127.0.0.1:43086)
0927 22:45:26.520345 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:26.520408 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:26.520646 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:26.522725 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:26.522733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:26.522771 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:26.523667 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.523679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.526002 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.526008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.528766 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.528779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.531170 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.531175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.531716 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.531755 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:45:26.531874 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:45:26.532027 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":420,"thread_start_us":3795,"threads_started":1}
I20260927 22:45:26.532680 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.519790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.1.53.65:45859)
0927 22:45:26.521420 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.521433 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.522362 (+ 929us) server_negotiation.cc:407] Connection header received
0927 22:45:26.522668 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.522677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.522940 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.523133 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:26.524199 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.525825 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.529340 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.530329 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.530493 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.530753 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.532155 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.532359 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404,"thread_start_us":1217,"threads_started":1}
I20260927 22:45:26.533258 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.521638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.1.53.66:34493)
0927 22:45:26.522700 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.522712 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.523618 (+ 906us) server_negotiation.cc:407] Connection header received
0927 22:45:26.523941 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.523950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.524181 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.524375 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:26.525095 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.526276 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.529255 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.530649 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.530780 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.530980 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.532703 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.532892 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":696,"threads_started":1}
I20260927 22:45:26.532173 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.521405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:34493 (local address 127.0.0.1:51102)
0927 22:45:26.523493 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:26.523513 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:26.523732 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:26.524653 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:26.524656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:26.524675 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:26.524938 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.524943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.526445 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.526450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.529064 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.529079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.531168 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.531175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.531428 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.531459 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:45:26.531554 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:45:26.531702 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"mutex_wait_us":192,"thread_start_us":554,"threads_started":1}
I20260927 22:45:26.535391 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.519568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.1.53.67:35029)
0927 22:45:26.521990 (+ 2422us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.522000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.522037 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:45:26.522147 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.522155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.522468 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.523177 (+ 709us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:26.524085 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.525837 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.528940 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.530888 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.531444 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.531690 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.532133 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.532354 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2227,"thread_start_us":2064,"threads_started":1}
I20260927 22:45:26.537614 19762 tablet_service.cc:1467] Processing CreateTablet for tablet 17b29b57673c4db9a59e72a1ff220a3b (DEFAULT_TABLE table=test-table [id=08390cb669f242eaaaa414fe5695e03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:26.537614 19628 tablet_service.cc:1467] Processing CreateTablet for tablet 17b29b57673c4db9a59e72a1ff220a3b (DEFAULT_TABLE table=test-table [id=08390cb669f242eaaaa414fe5695e03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:26.538667 19896 tablet_service.cc:1467] Processing CreateTablet for tablet 17b29b57673c4db9a59e72a1ff220a3b (DEFAULT_TABLE table=test-table [id=08390cb669f242eaaaa414fe5695e03a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:26.539127 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b29b57673c4db9a59e72a1ff220a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:26.539146 19762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b29b57673c4db9a59e72a1ff220a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:26.539831 19896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b29b57673c4db9a59e72a1ff220a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:26.552484 19980 tablet_bootstrap.cc:492] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Bootstrap starting.
I20260927 22:45:26.556216 19981 tablet_bootstrap.cc:492] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Bootstrap starting.
I20260927 22:45:26.557467 19980 tablet_bootstrap.cc:654] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:26.559006 19980 log.cc:824] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:26.560606 19981 tablet_bootstrap.cc:654] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:26.561544 19982 tablet_bootstrap.cc:492] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Bootstrap starting.
I20260927 22:45:26.562049 19981 log.cc:824] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:26.563236 19980 tablet_bootstrap.cc:492] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: No bootstrap required, opened a new log
I20260927 22:45:26.563572 19980 ts_tablet_manager.cc:1397] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260927 22:45:26.565974 19982 tablet_bootstrap.cc:654] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:26.567731 19982 log.cc:824] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:26.570801 19982 tablet_bootstrap.cc:492] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: No bootstrap required, opened a new log
I20260927 22:45:26.571038 19981 tablet_bootstrap.cc:492] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: No bootstrap required, opened a new log
I20260927 22:45:26.571177 19982 ts_tablet_manager.cc:1397] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260927 22:45:26.571326 19981 ts_tablet_manager.cc:1397] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260927 22:45:26.573164 19980 raft_consensus.cc:348] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.573683 19980 raft_consensus.cc:374] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:26.573787 19980 raft_consensus.cc:729] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478851c25e5e40d6b98bc102261c20a0, State: Initialized, Role: FOLLOWER
I20260927 22:45:26.574483 19980 consensus_queue.cc:260] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.576972 19961 heartbeater.cc:503] Master 127.1.53.126:37791 was elected leader, sending a full tablet report...
I20260927 22:45:26.577905 19980 ts_tablet_manager.cc:1428] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260927 22:45:26.581663 19981 raft_consensus.cc:348] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.582206 19981 raft_consensus.cc:374] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:26.582326 19981 raft_consensus.cc:729] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e6997c770644159fafb2835fa5fe07, State: Initialized, Role: FOLLOWER
I20260927 22:45:26.583003 19981 consensus_queue.cc:260] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.583039 19982 raft_consensus.cc:348] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.583474 19982 raft_consensus.cc:374] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:26.583591 19982 raft_consensus.cc:729] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d162552dbd3244b58713269f85ace912, State: Initialized, Role: FOLLOWER
I20260927 22:45:26.584362 19827 heartbeater.cc:503] Master 127.1.53.126:37791 was elected leader, sending a full tablet report...
I20260927 22:45:26.584364 19982 consensus_queue.cc:260] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.586392 19981 ts_tablet_manager.cc:1428] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 22:45:26.586396 19982 ts_tablet_manager.cc:1428] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260927 22:45:26.618268 19828 tablet.cc:2369] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:45:26.689932 19962 tablet.cc:2369] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:45:26.691627 19987 raft_consensus.cc:482] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:26.691951 19987 raft_consensus.cc:504] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.693897 19987 leader_election.cc:288] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 478851c25e5e40d6b98bc102261c20a0 (127.1.53.67:35029), d162552dbd3244b58713269f85ace912 (127.1.53.65:45859)
I20260927 22:45:26.700906 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.694300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:59195 (local address 127.1.53.67:35029)
0927 22:45:26.694565 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.694575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.695310 (+ 735us) server_negotiation.cc:407] Connection header received
0927 22:45:26.695602 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.695612 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.695675 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.695784 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:26.696366 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.697120 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.699041 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.699817 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.699984 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.700120 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.700492 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.700625 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:45:26.700975 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.694491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:40229 (local address 127.1.53.65:45859)
0927 22:45:26.694735 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.694745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.695616 (+ 871us) server_negotiation.cc:407] Connection header received
0927 22:45:26.695814 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.695824 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.695884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.695994 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:26.696641 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.697360 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.699154 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.699927 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.700099 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.700213 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.700576 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.700711 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:45:26.701059 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.694128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:35029 (local address 127.1.53.66:59195)
0927 22:45:26.695168 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:26.695188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:26.695407 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:26.695803 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:26.695808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:26.695833 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:26.696178 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.696188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.697265 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.697272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.698833 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.698846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.700081 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.700088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.700323 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.700353 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:45:26.700462 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:45:26.700622 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":1,"thread_start_us":636,"threads_started":1}
I20260927 22:45:26.701473 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.694329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:45859 (local address 127.1.53.66:40229)
0927 22:45:26.695494 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:26.695507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:26.695669 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:26.696023 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:26.696027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:26.696055 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:26.696488 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.696497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.697512 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.697517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.698986 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:26.698999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.700181 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.700188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:26.700404 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.700428 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:45:26.700546 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:45:26.700679 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":762,"threads_started":1}
I20260927 22:45:26.702296 19916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b29b57673c4db9a59e72a1ff220a3b" candidate_uuid: "05e6997c770644159fafb2835fa5fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478851c25e5e40d6b98bc102261c20a0" is_pre_election: true
I20260927 22:45:26.702620 19648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b29b57673c4db9a59e72a1ff220a3b" candidate_uuid: "05e6997c770644159fafb2835fa5fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d162552dbd3244b58713269f85ace912" is_pre_election: true
I20260927 22:45:26.702829 19916 raft_consensus.cc:2454] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6997c770644159fafb2835fa5fe07 in term 0.
I20260927 22:45:26.703174 19648 raft_consensus.cc:2454] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e6997c770644159fafb2835fa5fe07 in term 0.
I20260927 22:45:26.703724 19716 leader_election.cc:302] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e6997c770644159fafb2835fa5fe07, 478851c25e5e40d6b98bc102261c20a0; no voters:
I20260927 22:45:26.704205 19987 raft_consensus.cc:2781] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:45:26.704389 19987 raft_consensus.cc:482] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:45:26.704509 19987 raft_consensus.cc:3037] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:26.706992 19987 raft_consensus.cc:504] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.707683 19987 leader_election.cc:288] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [CANDIDATE]: Term 1 election: Requested vote from peers 478851c25e5e40d6b98bc102261c20a0 (127.1.53.67:35029), d162552dbd3244b58713269f85ace912 (127.1.53.65:45859)
I20260927 22:45:26.708482 19916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b29b57673c4db9a59e72a1ff220a3b" candidate_uuid: "05e6997c770644159fafb2835fa5fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478851c25e5e40d6b98bc102261c20a0"
I20260927 22:45:26.708540 19648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b29b57673c4db9a59e72a1ff220a3b" candidate_uuid: "05e6997c770644159fafb2835fa5fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d162552dbd3244b58713269f85ace912"
I20260927 22:45:26.708758 19648 raft_consensus.cc:3037] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:26.708760 19916 raft_consensus.cc:3037] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:26.712219 19648 raft_consensus.cc:2454] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6997c770644159fafb2835fa5fe07 in term 1.
I20260927 22:45:26.712265 19916 raft_consensus.cc:2454] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e6997c770644159fafb2835fa5fe07 in term 1.
I20260927 22:45:26.712838 19717 leader_election.cc:302] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e6997c770644159fafb2835fa5fe07, d162552dbd3244b58713269f85ace912; no voters:
I20260927 22:45:26.713497 19987 raft_consensus.cc:2781] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:26.714011 19987 raft_consensus.cc:686] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 1 LEADER]: Becoming Leader. State: Replica: 05e6997c770644159fafb2835fa5fe07, State: Running, Role: LEADER
I20260927 22:45:26.714867 19987 consensus_queue.cc:237] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } }
I20260927 22:45:26.719964 19504 catalog_manager.cc:5495] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e6997c770644159fafb2835fa5fe07 (127.1.53.66). New cstate: current_term: 1 leader_uuid: "05e6997c770644159fafb2835fa5fe07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e6997c770644159fafb2835fa5fe07" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 34493 } health_report { overall_health: HEALTHY } } }
I20260927 22:45:26.778240 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.772344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.1.53.66:34493)
0927 22:45:26.772584 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.772590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.772635 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:45:26.772916 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.772920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.773001 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.773201 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:45:26.773609 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.774345 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.775784 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.776215 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.776409 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.777062 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:45:26.777629 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:45:26.777846 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.777932 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.777981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:45:26.787134 19648 raft_consensus.cc:1261] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 1 FOLLOWER]: Refusing update from remote peer 05e6997c770644159fafb2835fa5fe07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:26.787196 19916 raft_consensus.cc:1261] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 1 FOLLOWER]: Refusing update from remote peer 05e6997c770644159fafb2835fa5fe07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:26.788290 19991 consensus_queue.cc:1035] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "478851c25e5e40d6b98bc102261c20a0" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:26.788713 19987 consensus_queue.cc:1035] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d162552dbd3244b58713269f85ace912" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 45859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:26.804579 19995 mvcc.cc:205] Tried to move back new op lower bound from 7334089223309094912 to 7334089223028293632. Current Snapshot: MvccSnapshot[applied={T|T < 7334089223309094912}]
I20260927 22:45:26.820900 19503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:26.821237 19503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:26.825261 19503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8584de1d3e0e4dce9dbd0b7022a18913: Sending DeleteTablet for 3 replicas of tablet 17b29b57673c4db9a59e72a1ff220a3b
I20260927 22:45:26.826583 19896 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b29b57673c4db9a59e72a1ff220a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:26 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:43086
I20260927 22:45:26.826681 19628 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b29b57673c4db9a59e72a1ff220a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:26 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:41896
I20260927 22:45:26.826700 19762 tablet_service.cc:1514] Processing DeleteTablet for tablet 17b29b57673c4db9a59e72a1ff220a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:26 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:51102
I20260927 22:45:26.832195 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19565
I20260927 22:45:26.841837 20005 tablet_replica.cc:330] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: stopping tablet replica
I20260927 22:45:26.842708 20005 raft_consensus.cc:2229] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:26.844918 20005 raft_consensus.cc:2258] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:26.846554 20007 tablet_replica.cc:330] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: stopping tablet replica
I20260927 22:45:26.847044 20007 raft_consensus.cc:2229] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:45:26.847738 20007 raft_consensus.cc:2258] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:26.847967 20005 ts_tablet_manager.cc:1905] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:26.852806 20007 ts_tablet_manager.cc:1905] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:26.853009 20006 tablet_replica.cc:330] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: stopping tablet replica
I20260927 22:45:26.853647 20006 raft_consensus.cc:2229] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:26.854282 20006 raft_consensus.cc:2258] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:26.856781 20006 ts_tablet_manager.cc:1905] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:26.860883 20007 ts_tablet_manager.cc:1918] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:26.861090 20007 log.cc:1196] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/17b29b57673c4db9a59e72a1ff220a3b
I20260927 22:45:26.861686 20007 ts_tablet_manager.cc:1939] T 17b29b57673c4db9a59e72a1ff220a3b P 05e6997c770644159fafb2835fa5fe07: Deleting consensus metadata
Sep 27 22:45:26 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:45:26.864763 19491 catalog_manager.cc:4841] TS 05e6997c770644159fafb2835fa5fe07 (127.1.53.66:34493): tablet 17b29b57673c4db9a59e72a1ff220a3b (table test-table [id=08390cb669f242eaaaa414fe5695e03a]) successfully deleted
I20260927 22:45:26.864907 20005 ts_tablet_manager.cc:1918] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:26.865238 20005 log.cc:1196] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/17b29b57673c4db9a59e72a1ff220a3b
I20260927 22:45:26.865836 20005 ts_tablet_manager.cc:1939] T 17b29b57673c4db9a59e72a1ff220a3b P 478851c25e5e40d6b98bc102261c20a0: Deleting consensus metadata
I20260927 22:45:26.867797 19489 catalog_manager.cc:4841] TS 478851c25e5e40d6b98bc102261c20a0 (127.1.53.67:35029): tablet 17b29b57673c4db9a59e72a1ff220a3b (table test-table [id=08390cb669f242eaaaa414fe5695e03a]) successfully deleted
I20260927 22:45:26.871587 20006 ts_tablet_manager.cc:1918] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:26.871863 20006 log.cc:1196] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/17b29b57673c4db9a59e72a1ff220a3b
I20260927 22:45:26.872802 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.833811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.1.53.65:45859)
0927 22:45:26.834075 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.834086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.848866 (+ 14780us) server_negotiation.cc:407] Connection header received
0927 22:45:26.849220 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.849227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.849359 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.849497 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:26.851586 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.852956 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.855856 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.856657 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.856779 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:26.863264 (+ 6485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:26.863286 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:26.863306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:26.863358 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:26.868484 (+ 5126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:26.869656 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:26.869668 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:26.869677 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:26.869891 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:26.870447 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:26.870465 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:26.870469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:26.870996 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:26.871312 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:26.872020 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 22:45:26.872348 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:45:26.872486 20006 ts_tablet_manager.cc:1939] T 17b29b57673c4db9a59e72a1ff220a3b P d162552dbd3244b58713269f85ace912: Deleting consensus metadata
I20260927 22:45:26.873910 19688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:26.974844 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19565
I20260927 22:45:26.974860 19490 catalog_manager.cc:4841] TS d162552dbd3244b58713269f85ace912 (127.1.53.65:45859): tablet 17b29b57673c4db9a59e72a1ff220a3b (table test-table [id=08390cb669f242eaaaa414fe5695e03a]) successfully deleted
I20260927 22:45:26.993150 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19699
Sep 27 22:45:26 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:45:27.013041 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:26.994028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.1.53.66:34493)
0927 22:45:26.994257 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:26.994263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:26.994287 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:45:26.994461 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:26.994468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:26.994562 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:26.994692 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:26.996071 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.997481 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.998767 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:26.999533 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:26.999681 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:27.004779 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:27.004802 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:27.004826 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:27.004877 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:27.009659 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:27.010570 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:27.010582 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:27.010586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:27.010793 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:27.011187 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:27.011193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:27.011197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:27.011741 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:27.012115 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:27.012325 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:45:27.012592 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:45:27.013823 19822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:27.131493 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19699
I20260927 22:45:27.150372 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19833
Sep 27 22:45:27 dist-test-slave-pp80 krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:45:27.166568 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:27.151208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.1.53.67:35029)
0927 22:45:27.151384 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:27.151389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:27.151407 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:45:27.151480 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:27.151484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:27.151558 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:27.151662 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:27.153117 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:27.153946 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:27.155388 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:27.155783 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:27.155883 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:27.160576 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:27.160598 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:27.160615 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:27.160657 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:27.163686 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:27.164685 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:27.164693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:27.164695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:27.164839 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:27.165199 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:27.165202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:27.165204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:27.165558 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:27.165777 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:27.166117 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 22:45:27.166289 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:45:27.167238 19956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:27.260180 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19833
I20260927 22:45:27.277241 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 19472
I20260927 22:45:27.278548 19534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:27.393553 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 19472
2026-09-27T22:45:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4589 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:27Z Disabled control of system clock
I20260927 22:45:27.474256 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:36359
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:45:27.821835 20016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:27.822156 20016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:27.822242 20016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:27.829707 20016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:27.829829 20016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:27.829867 20016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:27.829922 20016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:27.829968 20016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:27.839919 20016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:36359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:27.841355 20016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:27.843128 20016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:27.855899 20024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:27.855965 20022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:27.855835 20021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:27.856729 20016 server_base.cc:1034] running on GCE node
I20260927 22:45:27.857774 20016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:27.859485 20016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:27.860702 20016 hybrid_clock.cc:648] HybridClock initialized: now 1790549127860644 us; error 50 us; skew 500 ppm
I20260927 22:45:27.861192 20016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:27.869752 20016 webserver.cc:460] Webserver started at http://127.1.53.126:37489/ using document root <none> and password file <none>
I20260927 22:45:27.870673 20016 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:27.870779 20016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:27.871146 20016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:27.874459 20016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc"
format_stamp: "Formatted at 2026-09-27 22:45:27 on dist-test-slave-pp80"
server_key: "dd105d4a3199754023e3704fa88280ae"
server_key_iv: "caf9a89f1a320be1e39f8b1ea16c4c47"
server_key_version: "encryptionkey@0"
I20260927 22:45:27.875379 20016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc"
format_stamp: "Formatted at 2026-09-27 22:45:27 on dist-test-slave-pp80"
server_key: "dd105d4a3199754023e3704fa88280ae"
server_key_iv: "caf9a89f1a320be1e39f8b1ea16c4c47"
server_key_version: "encryptionkey@0"
I20260927 22:45:27.881309 20016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:45:27.885092 20030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:27.886736 20016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:27.886942 20016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc"
format_stamp: "Formatted at 2026-09-27 22:45:27 on dist-test-slave-pp80"
server_key: "dd105d4a3199754023e3704fa88280ae"
server_key_iv: "caf9a89f1a320be1e39f8b1ea16c4c47"
server_key_version: "encryptionkey@0"
I20260927 22:45:27.887190 20016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:27.904069 20016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:27.905027 20016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:27.905347 20016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:27.925668 20016 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:36359
I20260927 22:45:27.925684 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:36359 every 8 connection(s)
I20260927 22:45:27.927129 20016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:27.930615 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20016
I20260927 22:45:27.930812 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:27.931274 1237 external_mini_cluster.cc:1428] Setting key f73a77601bb35f6a09c95a6582a8aa84
I20260927 22:45:27.931615 20082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:27.945988 20082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc: Bootstrap starting.
I20260927 22:45:27.948385 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:27.933098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.1.53.126:36359)
0927 22:45:27.934166 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:27.934178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:27.936882 (+ 2704us) server_negotiation.cc:407] Connection header received
0927 22:45:27.939217 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:27.939241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:27.939794 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:27.940279 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:27.940854 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:27.942127 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:27.943536 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:27.944357 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:27.944561 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:27.945871 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:27.945879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:27.945895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:27.946553 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:27.946636 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:27.946660 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:27.946798 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:27.947027 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 22:45:27.947359 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":412,"threads_started":1}
I20260927 22:45:27.949512 20082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:27.950590 20082 log.cc:824] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:27.953719 20082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc: No bootstrap required, opened a new log
I20260927 22:45:27.961762 20082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 36359 } }
I20260927 22:45:27.962222 20082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:27.962333 20082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da6c77ee4684d6a8ab5bbe57044fcdc, State: Initialized, Role: FOLLOWER
I20260927 22:45:27.962998 20082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 36359 } }
I20260927 22:45:27.963249 20082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:27.963409 20082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:27.963591 20082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:27.965884 20082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 36359 } }
I20260927 22:45:27.966446 20082 leader_election.cc:302] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da6c77ee4684d6a8ab5bbe57044fcdc; no voters:
I20260927 22:45:27.967144 20082 leader_election.cc:288] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:27.967309 20087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:27.968380 20087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [term 1 LEADER]: Becoming Leader. State: Replica: 8da6c77ee4684d6a8ab5bbe57044fcdc, State: Running, Role: LEADER
I20260927 22:45:27.968902 20087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 36359 } }
I20260927 22:45:27.969458 20082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:27.971794 20088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 36359 } } }
I20260927 22:45:27.972220 20088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:27.972112 20089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da6c77ee4684d6a8ab5bbe57044fcdc. Latest consensus state: current_term: 1 leader_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da6c77ee4684d6a8ab5bbe57044fcdc" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 36359 } } }
I20260927 22:45:27.972514 20089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:27.975116 20095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:27.980949 20095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:27.990615 20095 catalog_manager.cc:1347] Generated new cluster ID: 8865610f298e4045be8c076bc0b4b364
I20260927 22:45:27.990772 20095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:28.032027 20095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:28.034029 20095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:28.044910 20095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc: Generated new TSK 0
I20260927 22:45:28.046094 20095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:28.052953 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:36359
--builtin_ntp_servers=127.1.53.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:45:28.421718 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:28.422026 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:28.422179 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:28.430114 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:28.430254 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:28.430467 20106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:45:28.440174 20106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:28.441790 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:28.443756 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:28.458220 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:28.458206 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:28.458523 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:28.459066 20106 server_base.cc:1034] running on GCE node
I20260927 22:45:28.459782 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:28.461005 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:28.462281 20106 hybrid_clock.cc:648] HybridClock initialized: now 1790549128462241 us; error 38 us; skew 500 ppm
I20260927 22:45:28.462695 20106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:28.465437 20106 webserver.cc:460] Webserver started at http://127.1.53.65:40645/ using document root <none> and password file <none>
I20260927 22:45:28.466269 20106 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:28.466420 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:28.466773 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:28.470088 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "652dd937c895432098eb7a1f06cc698d"
format_stamp: "Formatted at 2026-09-27 22:45:28 on dist-test-slave-pp80"
server_key: "47e48c997ce49dead5deb18ad04f8d28"
server_key_iv: "ef73b6b427bca746c64b77c0033cdc0d"
server_key_version: "encryptionkey@0"
I20260927 22:45:28.471078 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "652dd937c895432098eb7a1f06cc698d"
format_stamp: "Formatted at 2026-09-27 22:45:28 on dist-test-slave-pp80"
server_key: "47e48c997ce49dead5deb18ad04f8d28"
server_key_iv: "ef73b6b427bca746c64b77c0033cdc0d"
server_key_version: "encryptionkey@0"
I20260927 22:45:28.476751 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:45:28.480644 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:28.482353 20106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:28.482574 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "652dd937c895432098eb7a1f06cc698d"
format_stamp: "Formatted at 2026-09-27 22:45:28 on dist-test-slave-pp80"
server_key: "47e48c997ce49dead5deb18ad04f8d28"
server_key_iv: "ef73b6b427bca746c64b77c0033cdc0d"
server_key_version: "encryptionkey@0"
I20260927 22:45:28.482829 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:28.500028 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:28.501008 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:28.501331 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:28.502400 20106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:28.504035 20106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:28.504179 20106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:28.504369 20106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:28.504464 20106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:28.538877 20106 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:42679
I20260927 22:45:28.538975 20232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:42679 every 8 connection(s)
I20260927 22:45:28.540360 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:45:28.543349 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20106
I20260927 22:45:28.543533 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:45:28.544019 1237 external_mini_cluster.cc:1428] Setting key 6dcea6b356ceb7c0fff49ba0fa65a702
I20260927 22:45:28.546931 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:36359
--builtin_ntp_servers=127.1.53.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:45:28.554677 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:28.543115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:36913 (local address 127.1.53.126:36359)
0927 22:45:28.543698 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:28.543706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:28.547286 (+ 3580us) server_negotiation.cc:407] Connection header received
0927 22:45:28.547344 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:28.547347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:28.547415 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:28.547548 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:28.548807 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:28.549533 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:28.551525 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:28.551986 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:28.552138 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:28.553805 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:28.553810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:28.553814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:28.553835 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:28.553884 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:28.553897 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:28.553977 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:28.554291 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:45:28.554379 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":230,"threads_started":1}
I20260927 22:45:28.555841 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:28.543893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:36359 (local address 127.1.53.65:36913)
0927 22:45:28.545105 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:28.545163 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:28.546888 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:28.547821 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:28.547830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:28.547930 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:28.548625 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:28.548644 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:28.549713 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:28.549724 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:28.551344 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:28.551357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:28.552322 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:28.552337 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:28.552558 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:28.553384 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:28.553390 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:28.553453 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:28.553575 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:28.554011 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:28.554024 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:28.554034 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:45:28.554253 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:45:28.554693 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":454,"threads_started":1}
I20260927 22:45:28.557662 20233 heartbeater.cc:347] Connected to a master server at 127.1.53.126:36359
I20260927 22:45:28.558023 20233 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:28.559152 20233 heartbeater.cc:511] Master 127.1.53.126:36359 requested a full tablet report, sending...
I20260927 22:45:28.561743 20047 ts_manager.cc:194] Registered new tserver with Master: 652dd937c895432098eb7a1f06cc698d (127.1.53.65:42679)
I20260927 22:45:28.563301 20047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.65:36913
W20260927 22:45:28.893778 20237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:28.894083 20237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:28.894239 20237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:28.901979 20237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:28.902104 20237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:28.902294 20237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:45:28.912108 20237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:28.913573 20237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:28.915441 20237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:28.929674 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:28.929713 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:28.930037 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:28.931293 20237 server_base.cc:1034] running on GCE node
I20260927 22:45:28.932017 20237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:28.933235 20237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:28.934494 20237 hybrid_clock.cc:648] HybridClock initialized: now 1790549128934423 us; error 65 us; skew 500 ppm
I20260927 22:45:28.934857 20237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:28.937752 20237 webserver.cc:460] Webserver started at http://127.1.53.66:46423/ using document root <none> and password file <none>
I20260927 22:45:28.938624 20237 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:28.938736 20237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:28.939069 20237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:28.942965 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "5399c625a3764b808fc86959a5696a38"
format_stamp: "Formatted at 2026-09-27 22:45:28 on dist-test-slave-pp80"
server_key: "0c839a1ccd2ed32d18d50e36b14c8028"
server_key_iv: "8349edcafdafa6044f5ec6c4f30e774b"
server_key_version: "encryptionkey@0"
I20260927 22:45:28.943929 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "5399c625a3764b808fc86959a5696a38"
format_stamp: "Formatted at 2026-09-27 22:45:28 on dist-test-slave-pp80"
server_key: "0c839a1ccd2ed32d18d50e36b14c8028"
server_key_iv: "8349edcafdafa6044f5ec6c4f30e774b"
server_key_version: "encryptionkey@0"
I20260927 22:45:28.949777 20237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:45:28.953737 20251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:28.955295 20237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:28.955498 20237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "5399c625a3764b808fc86959a5696a38"
format_stamp: "Formatted at 2026-09-27 22:45:28 on dist-test-slave-pp80"
server_key: "0c839a1ccd2ed32d18d50e36b14c8028"
server_key_iv: "8349edcafdafa6044f5ec6c4f30e774b"
server_key_version: "encryptionkey@0"
I20260927 22:45:28.955754 20237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:28.969390 20237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:28.970293 20237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:28.970536 20237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:28.971582 20237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:28.973227 20237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:28.973313 20237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:28.973390 20237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:28.973449 20237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:29.009742 20237 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:41195
I20260927 22:45:29.009779 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:41195 every 8 connection(s)
I20260927 22:45:29.011206 20237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:45:29.016651 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20237
I20260927 22:45:29.016839 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:45:29.017365 1237 external_mini_cluster.cc:1428] Setting key 26a9b036e704f90732ff241c9b66aa02
I20260927 22:45:29.020560 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:36359
--builtin_ntp_servers=127.1.53.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:45:29.026688 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.014063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:48275 (local address 127.1.53.126:36359)
0927 22:45:29.014301 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.014307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.015817 (+ 1510us) server_negotiation.cc:407] Connection header received
0927 22:45:29.018400 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.018406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.018514 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.018639 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:29.019965 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.020878 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.023450 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.023868 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.023965 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.025820 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:29.025825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:29.025828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:29.025873 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:29.025905 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:29.025918 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:29.026067 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.026350 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.026435 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:45:29.028213 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.014654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:36359 (local address 127.1.53.66:48275)
0927 22:45:29.015636 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.015679 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.018089 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.018903 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.018912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.019023 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:29.019778 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.019805 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.021037 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.021045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.023298 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.023315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.024113 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.024120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.024504 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.025392 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:29.025397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:29.025457 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:29.025567 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:29.026083 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:29.026097 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:29.026109 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.026314 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.026770 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":345,"threads_started":1}
I20260927 22:45:29.029834 20364 heartbeater.cc:347] Connected to a master server at 127.1.53.126:36359
I20260927 22:45:29.030189 20364 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:29.031080 20364 heartbeater.cc:511] Master 127.1.53.126:36359 requested a full tablet report, sending...
I20260927 22:45:29.032761 20047 ts_manager.cc:194] Registered new tserver with Master: 5399c625a3764b808fc86959a5696a38 (127.1.53.66:41195)
I20260927 22:45:29.034029 20047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.66:48275
W20260927 22:45:29.366299 20367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:29.366592 20367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:29.366677 20367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:29.374580 20367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:29.374698 20367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:29.374807 20367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:45:29.384085 20367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:29.385434 20367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:29.387336 20367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:29.402002 20373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:29.402099 20375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:29.402163 20372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:29.402879 20367 server_base.cc:1034] running on GCE node
I20260927 22:45:29.403565 20367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:29.404716 20367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:29.406025 20367 hybrid_clock.cc:648] HybridClock initialized: now 1790549129405890 us; error 96 us; skew 500 ppm
I20260927 22:45:29.406385 20367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:29.409027 20367 webserver.cc:460] Webserver started at http://127.1.53.67:33683/ using document root <none> and password file <none>
I20260927 22:45:29.409785 20367 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:29.409930 20367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:29.410286 20367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:29.413357 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "ac03964307484a5692d53706e3f561e7"
format_stamp: "Formatted at 2026-09-27 22:45:29 on dist-test-slave-pp80"
server_key: "beb0e2321877ff2f99c28d9a11622b97"
server_key_iv: "bb234a8371ae4e3ace90ae36308c94a6"
server_key_version: "encryptionkey@0"
I20260927 22:45:29.414160 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "ac03964307484a5692d53706e3f561e7"
format_stamp: "Formatted at 2026-09-27 22:45:29 on dist-test-slave-pp80"
server_key: "beb0e2321877ff2f99c28d9a11622b97"
server_key_iv: "bb234a8371ae4e3ace90ae36308c94a6"
server_key_version: "encryptionkey@0"
I20260927 22:45:29.419610 20367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:45:29.423390 20381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:29.425028 20367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:29.425240 20367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "ac03964307484a5692d53706e3f561e7"
format_stamp: "Formatted at 2026-09-27 22:45:29 on dist-test-slave-pp80"
server_key: "beb0e2321877ff2f99c28d9a11622b97"
server_key_iv: "bb234a8371ae4e3ace90ae36308c94a6"
server_key_version: "encryptionkey@0"
I20260927 22:45:29.425486 20367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:29.442122 20367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:29.443070 20367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:29.443404 20367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:29.444624 20367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:29.446321 20367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:29.446455 20367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:29.446596 20367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:29.446689 20367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:29.485893 20367 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:34957
I20260927 22:45:29.485965 20493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:34957 every 8 connection(s)
I20260927 22:45:29.487591 20367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:45:29.490041 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20367
I20260927 22:45:29.490279 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:45:29.490898 1237 external_mini_cluster.cc:1428] Setting key 949ac818325dd505b3e8a7b03b4801bd
I20260927 22:45:29.501958 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.490957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:48259 (local address 127.1.53.126:36359)
0927 22:45:29.491190 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.491197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.493074 (+ 1877us) server_negotiation.cc:407] Connection header received
0927 22:45:29.494867 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.494872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.494959 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.495103 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:29.496412 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.497227 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.498866 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.499280 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.499430 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.501025 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:29.501034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:29.501041 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:29.501065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:29.501108 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:29.501121 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:29.501231 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.501493 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.501598 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:45:29.503384 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.491626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:36359 (local address 127.1.53.67:48259)
0927 22:45:29.492867 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.492912 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.494586 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.495435 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.495445 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.495546 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:29.496168 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.496186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.497408 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.497413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.498701 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.498714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.499528 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.499536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.499783 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.500657 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:29.500660 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:29.500704 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:29.500789 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:29.501229 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:29.501241 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:29.501252 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.501452 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.501892 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":454,"threads_started":1}
I20260927 22:45:29.505208 20494 heartbeater.cc:347] Connected to a master server at 127.1.53.126:36359
I20260927 22:45:29.505571 20494 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:29.506443 20494 heartbeater.cc:511] Master 127.1.53.126:36359 requested a full tablet report, sending...
I20260927 22:45:29.508387 20047 ts_manager.cc:194] Registered new tserver with Master: ac03964307484a5692d53706e3f561e7 (127.1.53.67:34957)
I20260927 22:45:29.509248 20047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.67:48259
I20260927 22:45:29.519531 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:45:29.530011 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.523918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.1.53.126:36359)
0927 22:45:29.524186 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.524195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.524442 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:45:29.524560 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.524564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.524645 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.524743 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:29.525216 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.525991 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.527411 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.527863 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.527972 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.529245 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:29.529250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:29.529254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:29.529272 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:29.529305 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:29.529317 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:29.529452 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.529622 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.529730 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:45:29.534165 20047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:45:29.541929 20047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:45:29.568050 20233 heartbeater.cc:503] Master 127.1.53.126:36359 was elected leader, sending a full tablet report...
I20260927 22:45:29.581555 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.555270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41578)
0927 22:45:29.571301 (+ 16031us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.571326 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.571447 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.572449 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.572459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.572530 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:29.573250 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.573272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.575811 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.575816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.578432 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.578444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.580460 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.580467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.580725 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.580782 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.580918 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.581154 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15880,"mutex_wait_us":167,"thread_start_us":16994,"threads_started":1}
I20260927 22:45:29.581583 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.555870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.1.53.65:42679)
0927 22:45:29.571022 (+ 15152us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.571034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.571623 (+ 589us) server_negotiation.cc:407] Connection header received
0927 22:45:29.571733 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.571743 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.572037 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.573774 (+ 1737us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:29.574055 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.575655 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.578643 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.580266 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.580405 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.580712 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.580994 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.581202 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14927,"thread_start_us":14764,"threads_started":1}
I20260927 22:45:29.582428 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.555655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.1.53.66:41195)
0927 22:45:29.571805 (+ 16150us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.571814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.571839 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:45:29.571923 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.571929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.572124 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.572557 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:29.573773 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.575460 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.579423 (+ 3963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.580586 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.580723 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.580961 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.581654 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.582087 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15958,"thread_start_us":17814,"threads_started":1}
I20260927 22:45:29.582955 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.555171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:41195 (local address 127.0.0.1:39224)
0927 22:45:29.570803 (+ 15632us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.570871 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.571146 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.572843 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.572847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.572869 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:29.573362 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.573372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.575628 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.575634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.579200 (+ 3566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.579214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.581107 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.581114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.581262 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.581295 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.581380 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.581525 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15272,"mutex_wait_us":336,"thread_start_us":15885,"threads_started":1}
I20260927 22:45:29.584295 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.573852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:34957 (local address 127.0.0.1:46076)
0927 22:45:29.574481 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.574508 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.574630 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.576133 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.576139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.576167 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:29.576832 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.576848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.578904 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.578910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.581002 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.581017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.583518 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.583525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.583696 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.583727 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.583812 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.583935 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":146,"thread_start_us":339,"threads_started":1}
I20260927 22:45:29.584470 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.572404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.1.53.67:34957)
0927 22:45:29.575224 (+ 2820us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.575233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.575270 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:45:29.575508 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.575520 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.575826 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.576124 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:29.577118 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.578745 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.581254 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.582702 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.583185 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.583438 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.583931 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.584123 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2625,"thread_start_us":1095,"threads_started":1}
I20260927 22:45:29.586944 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 915e5e2e52614e588fbae600b9e26f3d (DEFAULT_TABLE table=test-table [id=f85e5baae82b4252ac5a583bdbde6765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:29.586817 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 915e5e2e52614e588fbae600b9e26f3d (DEFAULT_TABLE table=test-table [id=f85e5baae82b4252ac5a583bdbde6765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:29.588506 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915e5e2e52614e588fbae600b9e26f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:29.588196 20429 tablet_service.cc:1467] Processing CreateTablet for tablet 915e5e2e52614e588fbae600b9e26f3d (DEFAULT_TABLE table=test-table [id=f85e5baae82b4252ac5a583bdbde6765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:29.588997 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915e5e2e52614e588fbae600b9e26f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:29.589658 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915e5e2e52614e588fbae600b9e26f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:29.601307 20512 tablet_bootstrap.cc:492] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: Bootstrap starting.
I20260927 22:45:29.604981 20512 tablet_bootstrap.cc:654] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:29.605300 20513 tablet_bootstrap.cc:492] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Bootstrap starting.
I20260927 22:45:29.606686 20512 log.cc:824] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:29.610203 20513 tablet_bootstrap.cc:654] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:29.611931 20514 tablet_bootstrap.cc:492] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Bootstrap starting.
I20260927 22:45:29.612047 20513 log.cc:824] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:29.616025 20514 tablet_bootstrap.cc:654] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:29.616267 20512 tablet_bootstrap.cc:492] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: No bootstrap required, opened a new log
I20260927 22:45:29.616636 20512 ts_tablet_manager.cc:1397] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260927 22:45:29.616948 20513 tablet_bootstrap.cc:492] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: No bootstrap required, opened a new log
I20260927 22:45:29.617304 20513 ts_tablet_manager.cc:1397] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:45:29.617838 20514 log.cc:824] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:29.620913 20514 tablet_bootstrap.cc:492] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: No bootstrap required, opened a new log
I20260927 22:45:29.621294 20514 ts_tablet_manager.cc:1397] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:45:29.628226 20513 raft_consensus.cc:348] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.628227 20512 raft_consensus.cc:348] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.628880 20512 raft_consensus.cc:374] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:29.629015 20513 raft_consensus.cc:374] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:29.629038 20512 raft_consensus.cc:729] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652dd937c895432098eb7a1f06cc698d, State: Initialized, Role: FOLLOWER
I20260927 22:45:29.629194 20513 raft_consensus.cc:729] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5399c625a3764b808fc86959a5696a38, State: Initialized, Role: FOLLOWER
I20260927 22:45:29.629894 20512 consensus_queue.cc:260] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.629894 20513 consensus_queue.cc:260] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.630369 20514 raft_consensus.cc:348] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.631304 20364 heartbeater.cc:503] Master 127.1.53.126:36359 was elected leader, sending a full tablet report...
I20260927 22:45:29.632165 20513 ts_tablet_manager.cc:1428] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260927 22:45:29.632468 20512 ts_tablet_manager.cc:1428] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260927 22:45:29.631007 20514 raft_consensus.cc:374] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:29.632823 20514 raft_consensus.cc:729] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac03964307484a5692d53706e3f561e7, State: Initialized, Role: FOLLOWER
I20260927 22:45:29.633534 20514 consensus_queue.cc:260] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.634682 20494 heartbeater.cc:503] Master 127.1.53.126:36359 was elected leader, sending a full tablet report...
I20260927 22:45:29.635387 20514 ts_tablet_manager.cc:1428] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
W20260927 22:45:29.739777 20495 tablet.cc:2369] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:45:29.763494 20365 tablet.cc:2369] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:45:29.792937 20234 tablet.cc:2369] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:45:29.927567 20520 raft_consensus.cc:482] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:29.927939 20520 raft_consensus.cc:504] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.930147 20520 leader_election.cc:288] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5399c625a3764b808fc86959a5696a38 (127.1.53.66:41195), 652dd937c895432098eb7a1f06cc698d (127.1.53.65:42679)
I20260927 22:45:29.937269 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.930427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.1.53.67:49105)
0927 22:45:29.930757 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.930782 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.930943 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.931386 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.931394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.931425 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:29.931659 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.931671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.933132 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.933138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.934986 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.935003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.936504 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.936510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.936697 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.936732 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.936819 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.936954 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":156}
I20260927 22:45:29.937299 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.930834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:49105 (local address 127.1.53.65:42679)
0927 22:45:29.931027 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.931033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.931054 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:45:29.931130 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.931147 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.931226 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.931316 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:29.931846 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.932969 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.935168 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.936327 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.936460 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.936582 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.936871 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.937028 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:45:29.938225 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.930848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:57229 (local address 127.1.53.66:41195)
0927 22:45:29.931236 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:29.931245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:29.931268 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:45:29.931487 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:29.931492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:29.931601 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:29.931672 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:29.932385 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.933229 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.935763 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.937006 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.937138 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.937247 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:29.937834 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 22:45:29.937969 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260927 22:45:29.938319 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:29.930498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:41195 (local address 127.1.53.67:57229)
0927 22:45:29.931186 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:29.931203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:29.931309 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:29.931879 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:29.931882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:29.931903 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:29.932179 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.932187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.933395 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.933401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.935552 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:29.935568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:29.937361 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:29.937367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:29.937620 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:29.937666 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:45:29.937789 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:45:29.937946 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":29,"thread_start_us":911,"threads_started":1}
I20260927 22:45:29.938804 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915e5e2e52614e588fbae600b9e26f3d" candidate_uuid: "ac03964307484a5692d53706e3f561e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652dd937c895432098eb7a1f06cc698d" is_pre_election: true
I20260927 22:45:29.939452 20188 raft_consensus.cc:2454] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac03964307484a5692d53706e3f561e7 in term 0.
I20260927 22:45:29.939534 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915e5e2e52614e588fbae600b9e26f3d" candidate_uuid: "ac03964307484a5692d53706e3f561e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5399c625a3764b808fc86959a5696a38" is_pre_election: true
I20260927 22:45:29.940028 20319 raft_consensus.cc:2454] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac03964307484a5692d53706e3f561e7 in term 0.
I20260927 22:45:29.940243 20383 leader_election.cc:302] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652dd937c895432098eb7a1f06cc698d, ac03964307484a5692d53706e3f561e7; no voters:
I20260927 22:45:29.940737 20520 raft_consensus.cc:2781] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:45:29.940886 20520 raft_consensus.cc:482] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:45:29.940976 20520 raft_consensus.cc:3037] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:29.944098 20520 raft_consensus.cc:504] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.944828 20520 leader_election.cc:288] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [CANDIDATE]: Term 1 election: Requested vote from peers 5399c625a3764b808fc86959a5696a38 (127.1.53.66:41195), 652dd937c895432098eb7a1f06cc698d (127.1.53.65:42679)
I20260927 22:45:29.945429 20188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915e5e2e52614e588fbae600b9e26f3d" candidate_uuid: "ac03964307484a5692d53706e3f561e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652dd937c895432098eb7a1f06cc698d"
I20260927 22:45:29.945446 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915e5e2e52614e588fbae600b9e26f3d" candidate_uuid: "ac03964307484a5692d53706e3f561e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5399c625a3764b808fc86959a5696a38"
I20260927 22:45:29.945693 20188 raft_consensus.cc:3037] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:29.945706 20319 raft_consensus.cc:3037] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:29.948343 20319 raft_consensus.cc:2454] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac03964307484a5692d53706e3f561e7 in term 1.
I20260927 22:45:29.948772 20188 raft_consensus.cc:2454] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac03964307484a5692d53706e3f561e7 in term 1.
I20260927 22:45:29.948803 20385 leader_election.cc:302] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5399c625a3764b808fc86959a5696a38, ac03964307484a5692d53706e3f561e7; no voters:
I20260927 22:45:29.949173 20520 raft_consensus.cc:2781] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:29.949672 20520 raft_consensus.cc:686] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 1 LEADER]: Becoming Leader. State: Replica: ac03964307484a5692d53706e3f561e7, State: Running, Role: LEADER
I20260927 22:45:29.950217 20520 consensus_queue.cc:237] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } }
I20260927 22:45:29.955399 20047 catalog_manager.cc:5495] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac03964307484a5692d53706e3f561e7 (127.1.53.67). New cstate: current_term: 1 leader_uuid: "ac03964307484a5692d53706e3f561e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac03964307484a5692d53706e3f561e7" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 34957 } health_report { overall_health: HEALTHY } } }
I20260927 22:45:30.048453 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.042456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.1.53.67:34957)
0927 22:45:30.042624 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:30.042628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:30.043127 (+ 499us) server_negotiation.cc:407] Connection header received
0927 22:45:30.043288 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:30.043292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:30.043373 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:30.043505 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:45:30.043958 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.044696 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.046109 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.046532 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.046684 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:30.047236 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:45:30.047764 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:45:30.047979 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:30.048074 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:45:30.048121 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:45:30.056951 20188 raft_consensus.cc:1261] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d [term 1 FOLLOWER]: Refusing update from remote peer ac03964307484a5692d53706e3f561e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:30.056946 20319 raft_consensus.cc:1261] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 1 FOLLOWER]: Refusing update from remote peer ac03964307484a5692d53706e3f561e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:30.058009 20522 consensus_queue.cc:1035] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5399c625a3764b808fc86959a5696a38" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:30.058385 20520 consensus_queue.cc:1035] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "652dd937c895432098eb7a1f06cc698d" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:30.071554 20526 mvcc.cc:205] Tried to move back new op lower bound from 7334089236703039488 to 7334089236280524800. Current Snapshot: MvccSnapshot[applied={T|T < 7334089236703039488}]
I20260927 22:45:30.072377 20528 mvcc.cc:205] Tried to move back new op lower bound from 7334089236703039488 to 7334089236280524800. Current Snapshot: MvccSnapshot[applied={T|T < 7334089236703039488}]
I20260927 22:45:30.090798 20046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:30.091113 20046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:30.095114 20046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da6c77ee4684d6a8ab5bbe57044fcdc: Sending DeleteTablet for 3 replicas of tablet 915e5e2e52614e588fbae600b9e26f3d
I20260927 22:45:30.096628 20429 tablet_service.cc:1514] Processing DeleteTablet for tablet 915e5e2e52614e588fbae600b9e26f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:30 UTC) from {username='slave'} at 127.0.0.1:46076
I20260927 22:45:30.097036 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20106
I20260927 22:45:30.097410 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet 915e5e2e52614e588fbae600b9e26f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:30 UTC) from {username='slave'} at 127.0.0.1:39224
I20260927 22:45:30.098582 20538 tablet_replica.cc:330] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: stopping tablet replica
I20260927 22:45:30.098994 20538 raft_consensus.cc:2229] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:30.099608 20538 raft_consensus.cc:2258] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:30.100594 20168 tablet_service.cc:1514] Processing DeleteTablet for tablet 915e5e2e52614e588fbae600b9e26f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:30 UTC) from {username='slave'} at 127.0.0.1:41578
I20260927 22:45:30.101998 20538 ts_tablet_manager.cc:1905] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:30.107618 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.099363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.1.53.65:42679)
0927 22:45:30.099563 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:30.099583 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:30.099610 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:45:30.099675 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:30.099681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:30.099810 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:30.099932 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:30.100460 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.101650 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.103531 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.104151 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.104283 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:30.105845 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:30.105856 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:30.105873 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:30.106701 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:30.106766 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:30.106790 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:30.106886 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:30.107124 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:45:30.107253 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:45:30.108897 20228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:30.109119 20538 ts_tablet_manager.cc:1918] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:30.109351 20538 log.cc:1196] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/915e5e2e52614e588fbae600b9e26f3d
I20260927 22:45:30.110095 20538 ts_tablet_manager.cc:1939] T 915e5e2e52614e588fbae600b9e26f3d P 5399c625a3764b808fc86959a5696a38: Deleting consensus metadata
I20260927 22:45:30.112355 20034 catalog_manager.cc:4841] TS 5399c625a3764b808fc86959a5696a38 (127.1.53.66:41195): tablet 915e5e2e52614e588fbae600b9e26f3d (table test-table [id=f85e5baae82b4252ac5a583bdbde6765]) successfully deleted
I20260927 22:45:30.115334 20537 tablet_replica.cc:330] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: stopping tablet replica
I20260927 22:45:30.115993 20537 raft_consensus.cc:2229] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:45:30.116828 20537 raft_consensus.cc:2258] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:30.119179 20537 ts_tablet_manager.cc:1905] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:30.128428 20537 ts_tablet_manager.cc:1918] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:30.128676 20537 log.cc:1196] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/915e5e2e52614e588fbae600b9e26f3d
I20260927 22:45:30.129320 20537 ts_tablet_manager.cc:1939] T 915e5e2e52614e588fbae600b9e26f3d P ac03964307484a5692d53706e3f561e7: Deleting consensus metadata
I20260927 22:45:30.131233 20034 catalog_manager.cc:4841] TS ac03964307484a5692d53706e3f561e7 (127.1.53.67:34957): tablet 915e5e2e52614e588fbae600b9e26f3d (table test-table [id=f85e5baae82b4252ac5a583bdbde6765]) successfully deleted
I20260927 22:45:30.207971 20540 tablet_replica.cc:330] T 915e5e2e52614e588fbae600b9e26f3d P 652dd937c895432098eb7a1f06cc698d: stopping tablet replica
I20260927 22:45:30.208334 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20106
I20260927 22:45:30.232446 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20237
W20260927 22:45:30.232506 20032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:42679 (error 108)
I20260927 22:45:30.235111 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.234039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41600)
0927 22:45:30.234700 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.234871 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":450,"threads_started":1}
W20260927 22:45:30.235412 20032 catalog_manager.cc:4584] TS 652dd937c895432098eb7a1f06cc698d (127.1.53.65:42679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 915e5e2e52614e588fbae600b9e26f3d: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
I20260927 22:45:30.241696 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.233642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.1.53.66:41195)
0927 22:45:30.233857 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:30.233863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:30.233883 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:45:30.233952 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:30.233958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:30.234064 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:30.234198 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:30.234728 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.235789 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.238175 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.238617 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.238737 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:30.240012 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:30.240022 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:30.240037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:30.240797 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:30.240850 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:30.240872 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:30.240967 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:30.241178 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 22:45:30.241298 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:45:30.242369 20359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:30.289215 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.288640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41604)
0927 22:45:30.288903 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.289021 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 22:45:30.290524 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.290053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41610)
0927 22:45:30.290229 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.290334 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:45:30.340576 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20237
I20260927 22:45:30.351353 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.350757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41624)
0927 22:45:30.351029 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.351155 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:45:30.352795 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.352297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41630)
0927 22:45:30.352489 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.352597 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:45:30.359056 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20367
I20260927 22:45:30.366732 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.360002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.1.53.67:34957)
0927 22:45:30.360216 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:30.360222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:30.360281 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:45:30.360383 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:30.360389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:30.360505 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:30.360637 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:30.361056 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.361920 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.363338 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:30.363791 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:30.363945 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:30.365309 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:30.365318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:30.365333 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:30.365909 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:30.365971 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:30.365992 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:30.366111 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:30.366267 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:45:30.366369 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:45:30.367455 20489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:30.433513 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.432930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41636)
0927 22:45:30.433197 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.433313 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 22:45:30.434783 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:30.434261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:42679 (local address 127.0.0.1:41650)
0927 22:45:30.434505 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:30.434599 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:42679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 22:45:30.495179 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20367
I20260927 22:45:30.512701 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20016
I20260927 22:45:30.513926 20077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:30.629799 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20016
2026-09-27T22:45:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3233 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:30 dist-test-slave-pp80 krb5kdc[20549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:30 dist-test-slave-pp80 krb5kdc[20549](info): set up 2 sockets
Sep 27 22:45:30 dist-test-slave-pp80 krb5kdc[20549](info): commencing operation
krb5kdc: starting...
W20260927 22:45:31.772840 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:31 dist-test-slave-pp80 krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:45:32.044482 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42463
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:32.407687 20562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:32.408038 20562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:32.408219 20562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:32.416056 20562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:32.416218 20562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:32.416317 20562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:32.416416 20562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:32.416491 20562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:32.428681 20562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:32.430126 20562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:32.431478 20562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:32.444911 20567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:32.444890 20570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:32.445288 20568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:32.445899 20562 server_base.cc:1034] running on GCE node
I20260927 22:45:32.446779 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:32.448474 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:32.449710 20562 hybrid_clock.cc:648] HybridClock initialized: now 1790549132449643 us; error 66 us; skew 500 ppm
I20260927 22:45:32.450160 20562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:32 dist-test-slave-pp80 krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549132, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:32.453550 20562 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:45:32.455426 20562 webserver.cc:460] Webserver started at http://127.1.53.126:34753/ using document root <none> and password file <none>
I20260927 22:45:32.456117 20562 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:32.456230 20562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:32.456615 20562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:32.459753 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "8d48b5fc02d843738ac6e7373660a862"
format_stamp: "Formatted at 2026-09-27 22:45:32 on dist-test-slave-pp80"
server_key: "86abff80171c32d703e3d5f7973cc169"
server_key_iv: "f140576f1e2cc18643a49238e4db7ff9"
server_key_version: "encryptionkey@0"
I20260927 22:45:32.460637 20562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "8d48b5fc02d843738ac6e7373660a862"
format_stamp: "Formatted at 2026-09-27 22:45:32 on dist-test-slave-pp80"
server_key: "86abff80171c32d703e3d5f7973cc169"
server_key_iv: "f140576f1e2cc18643a49238e4db7ff9"
server_key_version: "encryptionkey@0"
I20260927 22:45:32.466266 20562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:45:32.470124 20577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:32.471632 20562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:45:32.471793 20562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "8d48b5fc02d843738ac6e7373660a862"
format_stamp: "Formatted at 2026-09-27 22:45:32 on dist-test-slave-pp80"
server_key: "86abff80171c32d703e3d5f7973cc169"
server_key_iv: "f140576f1e2cc18643a49238e4db7ff9"
server_key_version: "encryptionkey@0"
I20260927 22:45:32.471971 20562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:32.492271 20562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:32.497421 20562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:32.497786 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:32.518532 20562 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42463
I20260927 22:45:32.518554 20628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42463 every 8 connection(s)
I20260927 22:45:32.520032 20562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:32.521559 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20562
I20260927 22:45:32.521744 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:32.522207 1237 external_mini_cluster.cc:1428] Setting key ac81d5aa3d3618fd29c9ffddbd16eb43
I20260927 22:45:32.524775 20629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:45:32 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:32.538446 20629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862: Bootstrap starting.
I20260927 22:45:32.542111 20629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:32.543286 20629 log.cc:824] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:32.546733 20629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862: No bootstrap required, opened a new log
I20260927 22:45:32.547725 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:32.524292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.1.53.126:42463)
0927 22:45:32.525142 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:32.525158 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:32.525213 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:45:32.527664 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:32.527695 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:32.528268 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:32.528777 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:32.530238 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:32.532405 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:32.533758 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:32.534820 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:32.535021 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:32.539599 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:32.539611 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:32.539629 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:32.539672 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:32.542948 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:32.543878 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:32.543891 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:32.543902 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:32.544232 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:32.544774 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:32.544779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:32.544782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:32.545346 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:32.545731 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:32.546197 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 22:45:32.546657 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":348,"threads_started":1}
I20260927 22:45:32.555704 20629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b5fc02d843738ac6e7373660a862" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42463 } }
I20260927 22:45:32.556193 20629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:32.556373 20629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d48b5fc02d843738ac6e7373660a862, State: Initialized, Role: FOLLOWER
I20260927 22:45:32.557147 20629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b5fc02d843738ac6e7373660a862" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42463 } }
I20260927 22:45:32.557423 20629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:32.557588 20629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:32.557816 20629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:32.560225 20629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b5fc02d843738ac6e7373660a862" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42463 } }
I20260927 22:45:32.560843 20629 leader_election.cc:302] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d48b5fc02d843738ac6e7373660a862; no voters:
I20260927 22:45:32.561586 20629 leader_election.cc:288] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:32.561745 20634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:32.563141 20634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [term 1 LEADER]: Becoming Leader. State: Replica: 8d48b5fc02d843738ac6e7373660a862, State: Running, Role: LEADER
I20260927 22:45:32.563750 20634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b5fc02d843738ac6e7373660a862" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42463 } }
I20260927 22:45:32.564047 20629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:32.567240 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d48b5fc02d843738ac6e7373660a862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b5fc02d843738ac6e7373660a862" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42463 } } }
I20260927 22:45:32.567220 20636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d48b5fc02d843738ac6e7373660a862. Latest consensus state: current_term: 1 leader_uuid: "8d48b5fc02d843738ac6e7373660a862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d48b5fc02d843738ac6e7373660a862" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42463 } } }
I20260927 22:45:32.567814 20636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:32.567814 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:32.568888 20642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:32.575330 20642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:32.584385 20651 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:45:32.590276 20642 catalog_manager.cc:1347] Generated new cluster ID: c79764b66bbc4d56a25eed99c203681a
I20260927 22:45:32.590412 20642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:32.608942 20642 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:32.610368 20642 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:32.621378 20642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862: Generated new TSK 0
I20260927 22:45:32.622390 20642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
Entry for principal kudu/127.1.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab.
I20260927 22:45:32.691651 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42463
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:33.048385 20655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:33.048702 20655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:33.048866 20655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:33.056632 20655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:33.056749 20655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:33.056981 20655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:45:33.067396 20655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.65.keytab
--principal=kudu/127.1.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:33.068912 20655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:33.070328 20655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:33.084458 20660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:33.084558 20661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:33.084582 20663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:33.085733 20655 server_base.cc:1034] running on GCE node
I20260927 22:45:33.086488 20655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:33.087498 20655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:33.088732 20655 hybrid_clock.cc:648] HybridClock initialized: now 1790549133088673 us; error 36 us; skew 500 ppm
I20260927 22:45:33.089161 20655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:33 dist-test-slave-pp80 krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549133, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:33.104653 20655 init.cc:377] Logged in from keytab as kudu/127.1.53.65@KRBTEST.COM (short username kudu)
I20260927 22:45:33.107095 20655 webserver.cc:460] Webserver started at http://127.1.53.65:35975/ using document root <none> and password file <none>
I20260927 22:45:33.108078 20655 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:33.108218 20655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:33.108595 20655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:33.111662 20655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "cc16a6835d6e419d92b52a8bad00a3f1"
format_stamp: "Formatted at 2026-09-27 22:45:33 on dist-test-slave-pp80"
server_key: "74e7e02edcf5d7305e91337393c84a52"
server_key_iv: "cae8ac44a882b48d4b967b33ba51b1a9"
server_key_version: "encryptionkey@0"
I20260927 22:45:33.112581 20655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "cc16a6835d6e419d92b52a8bad00a3f1"
format_stamp: "Formatted at 2026-09-27 22:45:33 on dist-test-slave-pp80"
server_key: "74e7e02edcf5d7305e91337393c84a52"
server_key_iv: "cae8ac44a882b48d4b967b33ba51b1a9"
server_key_version: "encryptionkey@0"
I20260927 22:45:33.118156 20655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:45:33.122099 20670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:33.123587 20655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 22:45:33.123792 20655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "cc16a6835d6e419d92b52a8bad00a3f1"
format_stamp: "Formatted at 2026-09-27 22:45:33 on dist-test-slave-pp80"
server_key: "74e7e02edcf5d7305e91337393c84a52"
server_key_iv: "cae8ac44a882b48d4b967b33ba51b1a9"
server_key_version: "encryptionkey@0"
I20260927 22:45:33.123983 20655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:33.148880 20655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:33.153650 20655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:33.154004 20655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:33.155081 20655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:33.156672 20655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:33.156788 20655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:33.156946 20655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:33.157020 20655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:33.192440 20655 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:40565
I20260927 22:45:33.192533 20782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:40565 every 8 connection(s)
I20260927 22:45:33.193835 20655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:45:33.198582 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20655
I20260927 22:45:33.198815 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:45:33.199419 1237 external_mini_cluster.cc:1428] Setting key 5ecdca04f6dffd1a74bb1959b9e26078
Sep 27 22:45:33 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549133, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.65@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:33.219272 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:33.196793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:44359 (local address 127.1.53.126:42463)
0927 22:45:33.197274 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:33.197280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:33.198347 (+ 1067us) server_negotiation.cc:407] Connection header received
0927 22:45:33.200953 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:33.200958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:33.201021 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:33.201115 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:33.203297 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.204054 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.206679 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.207116 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.207232 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:33.212010 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:33.212015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:33.212018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:33.212054 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:33.215669 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:33.216613 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:33.216622 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:33.216628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:33.216791 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:33.217225 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:33.217233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:33.217239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:33.217625 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:33.217801 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:33.218518 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 22:45:33.218779 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":213,"threads_started":1}
I20260927 22:45:33.219983 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:33.197355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42463 (local address 127.1.53.65:44359)
0927 22:45:33.198085 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:33.198199 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:33.200635 (+ 2436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:33.201423 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:33.201433 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:33.202211 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:33.203087 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:33.203117 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.204295 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.204304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:33.206494 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:33.206512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.207374 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.207383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:33.207758 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:33.208613 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:33.208647 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:33.211729 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:33.215860 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:33.215868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:33.215885 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:33.216461 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:33.216905 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:33.216908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:33.216910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:33.217105 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:33.217753 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:33.217759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:33.218098 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 22:45:33.218355 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 22:45:33.218781 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":294,"threads_started":1}
I20260927 22:45:33.221712 20783 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42463
I20260927 22:45:33.222028 20783 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:33.222767 20783 heartbeater.cc:511] Master 127.1.53.126:42463 requested a full tablet report, sending...
I20260927 22:45:33.225117 20594 ts_manager.cc:194] Registered new tserver with Master: cc16a6835d6e419d92b52a8bad00a3f1 (127.1.53.65:40565)
I20260927 22:45:33.226589 20594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.65@KRBTEST.COM'} at 127.1.53.65:44359
WARNING: no policy specified for kudu/127.1.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
Entry for principal kudu/127.1.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab.
I20260927 22:45:33.263532 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42463
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:33.613870 20789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:33.614236 20789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:33.614406 20789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:33.622217 20789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:33.622385 20789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:33.622565 20789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:45:33.632984 20789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.66.keytab
--principal=kudu/127.1.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:33.634438 20789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:33.635864 20789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:33.650273 20794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:33.650256 20797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:33.650331 20795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:33.651041 20789 server_base.cc:1034] running on GCE node
I20260927 22:45:33.651784 20789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:33.652861 20789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:33.654091 20789 hybrid_clock.cc:648] HybridClock initialized: now 1790549133654039 us; error 47 us; skew 500 ppm
I20260927 22:45:33.654417 20789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:33 dist-test-slave-pp80 krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549133, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:33.657677 20789 init.cc:377] Logged in from keytab as kudu/127.1.53.66@KRBTEST.COM (short username kudu)
I20260927 22:45:33.659394 20789 webserver.cc:460] Webserver started at http://127.1.53.66:34627/ using document root <none> and password file <none>
I20260927 22:45:33.660135 20789 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:33.660305 20789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:33.660643 20789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:33.663625 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "a145bb6fff7540509a840dec27a33568"
format_stamp: "Formatted at 2026-09-27 22:45:33 on dist-test-slave-pp80"
server_key: "0b45725f51e79bc2f6fd2b4c77bc6f48"
server_key_iv: "88b4bf8b98d5f94cf5a2c800b72aa0db"
server_key_version: "encryptionkey@0"
I20260927 22:45:33.664584 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "a145bb6fff7540509a840dec27a33568"
format_stamp: "Formatted at 2026-09-27 22:45:33 on dist-test-slave-pp80"
server_key: "0b45725f51e79bc2f6fd2b4c77bc6f48"
server_key_iv: "88b4bf8b98d5f94cf5a2c800b72aa0db"
server_key_version: "encryptionkey@0"
I20260927 22:45:33.670468 20789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:45:33.674328 20804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:33.675848 20789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:45:33.676024 20789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "a145bb6fff7540509a840dec27a33568"
format_stamp: "Formatted at 2026-09-27 22:45:33 on dist-test-slave-pp80"
server_key: "0b45725f51e79bc2f6fd2b4c77bc6f48"
server_key_iv: "88b4bf8b98d5f94cf5a2c800b72aa0db"
server_key_version: "encryptionkey@0"
I20260927 22:45:33.676235 20789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:33.695497 20789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:33.700013 20789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:33.700400 20789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:33.701504 20789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:33.702900 20789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:33.702978 20789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:33.703138 20789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:33.703197 20789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:33.739279 20789 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:40603
I20260927 22:45:33.739336 20916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:40603 every 8 connection(s)
I20260927 22:45:33.740790 20789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:45:33.750718 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20789
I20260927 22:45:33.750916 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:45:33.751461 1237 external_mini_cluster.cc:1428] Setting key 216f58757bcdb1e8dcd701665d964562
Sep 27 22:45:33 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549133, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.66@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:33.764636 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:33.743571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:60847 (local address 127.1.53.126:42463)
0927 22:45:33.744002 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:33.744007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:33.745099 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 22:45:33.747026 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:33.747032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:33.747127 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:33.747253 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:33.748951 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.749683 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.751855 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.752558 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.752714 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:33.756639 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:33.756647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:33.756654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:33.756702 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:33.760442 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:33.761351 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:33.761360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:33.761366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:33.761539 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:33.762155 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:33.762163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:33.762169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:33.762580 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:33.762736 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:33.763917 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0927 22:45:33.764213 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":216,"threads_started":1}
I20260927 22:45:33.766310 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:33.744171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42463 (local address 127.1.53.66:60847)
0927 22:45:33.744928 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:33.744972 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:33.746736 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:33.747568 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:33.747581 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:33.748135 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:33.748775 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:33.748792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.749847 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.749851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:33.751699 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:33.751714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:33.752848 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:33.752856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:33.753089 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:33.753806 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:33.753835 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:33.756384 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:33.760625 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:33.760633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:33.760649 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:33.761191 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:33.761699 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:33.761703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:33.761705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:33.761929 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:33.762749 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:33.762759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:33.763407 (+ 648us) client_negotiation.cc:769] Sending connection context
0927 22:45:33.763790 (+ 383us) client_negotiation.cc:240] Negotiation successful
0927 22:45:33.764566 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":264,"threads_started":1}
I20260927 22:45:33.768198 20917 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42463
I20260927 22:45:33.768641 20917 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:33.769613 20917 heartbeater.cc:511] Master 127.1.53.126:42463 requested a full tablet report, sending...
I20260927 22:45:33.771477 20594 ts_manager.cc:194] Registered new tserver with Master: a145bb6fff7540509a840dec27a33568 (127.1.53.66:40603)
I20260927 22:45:33.772529 20594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.66@KRBTEST.COM'} at 127.1.53.66:60847
WARNING: no policy specified for kudu/127.1.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
Entry for principal kudu/127.1.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab.
I20260927 22:45:33.811992 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42463
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:34.151185 20923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:34.151526 20923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:34.151710 20923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:34.159126 20923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:34.159297 20923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:34.159487 20923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:45:34.169603 20923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.67.keytab
--principal=kudu/127.1.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:34.171190 20923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:34.172652 20923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:34.186902 20931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:34.186919 20929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:34.186914 20928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:34.187613 20923 server_base.cc:1034] running on GCE node
I20260927 22:45:34.188556 20923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:34.189658 20923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:34.190902 20923 hybrid_clock.cc:648] HybridClock initialized: now 1790549134190841 us; error 59 us; skew 500 ppm
I20260927 22:45:34.191257 20923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:34 dist-test-slave-pp80 krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549134, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:34.195302 20923 init.cc:377] Logged in from keytab as kudu/127.1.53.67@KRBTEST.COM (short username kudu)
I20260927 22:45:34.197096 20923 webserver.cc:460] Webserver started at http://127.1.53.67:35819/ using document root <none> and password file <none>
I20260927 22:45:34.197815 20923 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:34.197962 20923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:34.198280 20923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:34.201427 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "8c9b01ee31b94075b5cfbb8a4ac38827"
format_stamp: "Formatted at 2026-09-27 22:45:34 on dist-test-slave-pp80"
server_key: "9313ea691b14e4cccf6512c5889dd6ca"
server_key_iv: "59a385d4d42f792926251a183193ed5d"
server_key_version: "encryptionkey@0"
I20260927 22:45:34.202330 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "8c9b01ee31b94075b5cfbb8a4ac38827"
format_stamp: "Formatted at 2026-09-27 22:45:34 on dist-test-slave-pp80"
server_key: "9313ea691b14e4cccf6512c5889dd6ca"
server_key_iv: "59a385d4d42f792926251a183193ed5d"
server_key_version: "encryptionkey@0"
I20260927 22:45:34.207839 20923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:45:34.211496 20938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:34.213109 20923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:34.213301 20923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "8c9b01ee31b94075b5cfbb8a4ac38827"
format_stamp: "Formatted at 2026-09-27 22:45:34 on dist-test-slave-pp80"
server_key: "9313ea691b14e4cccf6512c5889dd6ca"
server_key_iv: "59a385d4d42f792926251a183193ed5d"
server_key_version: "encryptionkey@0"
I20260927 22:45:34.213532 20923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:34.230034 20783 heartbeater.cc:503] Master 127.1.53.126:42463 was elected leader, sending a full tablet report...
I20260927 22:45:34.240921 20923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:34.245779 20923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:34.246094 20923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:34.247144 20923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:34.248760 20923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:34.248837 20923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:34.248924 20923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:34.248967 20923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:34.288686 20923 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:37181
I20260927 22:45:34.288776 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:37181 every 8 connection(s)
I20260927 22:45:34.290040 20923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:45:34.298395 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 20923
I20260927 22:45:34.298619 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:45:34.299050 1237 external_mini_cluster.cc:1428] Setting key b939c043313ecee6e54f38efa2b7fce0
Sep 27 22:45:34 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549134, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.67@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:45:34.311617 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.292723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:34399 (local address 127.1.53.126:42463)
0927 22:45:34.293231 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.293236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.294017 (+ 781us) server_negotiation.cc:407] Connection header received
0927 22:45:34.296025 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.296032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.296153 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.296313 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:34.297969 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.298947 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.301187 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.301632 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.301785 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.305512 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:34.305518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:34.305522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:34.305558 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:34.308573 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.309344 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.309348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.309350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.309484 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.309957 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.309960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.309962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.310262 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:34.310390 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.311108 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.311321 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260927 22:45:34.312850 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.293116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:42463 (local address 127.1.53.67:34399)
0927 22:45:34.293839 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:34.293888 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:34.295727 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:34.296577 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:34.296586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:34.297152 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:34.297779 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.297799 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.299115 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.299120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.301011 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.301025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.301882 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.301889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.302219 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.302889 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:45:34.302915 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:34.305286 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:34.308721 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:34.308729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:34.308744 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:34.309222 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:34.309642 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:45:34.309646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:45:34.309648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:45:34.309845 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:45:34.310396 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:34.310403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:34.310758 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 22:45:34.310996 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 22:45:34.311447 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":282,"threads_started":1}
I20260927 22:45:34.314291 21051 heartbeater.cc:347] Connected to a master server at 127.1.53.126:42463
I20260927 22:45:34.314627 21051 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:34.315387 21051 heartbeater.cc:511] Master 127.1.53.126:42463 requested a full tablet report, sending...
I20260927 22:45:34.316884 20594 ts_manager.cc:194] Registered new tserver with Master: 8c9b01ee31b94075b5cfbb8a4ac38827 (127.1.53.67:37181)
I20260927 22:45:34.317700 20594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.53.67@KRBTEST.COM'} at 127.1.53.67:34399
I20260927 22:45:34.327082 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:45:34.344013 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.331467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.1.53.126:42463)
0927 22:45:34.331737 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.331747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.332081 (+ 334us) server_negotiation.cc:407] Connection header received
0927 22:45:34.332282 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.332292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.332375 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.332467 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:34.333890 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.334679 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.335943 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.336485 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.336585 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.338972 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:34.338977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:34.338980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:34.339009 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:34.341632 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.342253 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.342257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.342260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.342370 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.342688 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.342691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.342693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.343006 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:34.343195 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.343473 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.343670 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:45:34.348289 20594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:45:34.355185 20594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:45:34.391867 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.367730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:37181 (local address 127.0.0.1:56830)
0927 22:45:34.379888 (+ 12158us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:34.379931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:34.380152 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:34.383435 (+ 3283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:34.383448 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:34.383494 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:34.384008 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.384024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.386290 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.386296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.388877 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.388891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.390849 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.390855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.391075 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.391133 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:45:34.391286 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 22:45:34.391484 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11879,"thread_start_us":17079,"threads_started":1}
I20260927 22:45:34.391888 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.368791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.1.53.67:37181)
0927 22:45:34.382577 (+ 13786us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.382589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.382633 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:45:34.382766 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.382778 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.383129 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.383389 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:34.384350 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.386085 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.389083 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.390617 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.390779 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.391037 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.391351 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.391534 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13588,"thread_start_us":13429,"threads_started":1}
I20260927 22:45:34.393309 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.368198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:40603 (local address 127.0.0.1:45642)
0927 22:45:34.381742 (+ 13544us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:34.381770 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:34.381943 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:34.385617 (+ 3674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:34.385622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:34.385647 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:34.386045 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.386058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.387988 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.387997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.390103 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.390115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.392441 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.392449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.392649 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.392680 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:45:34.392769 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:45:34.392894 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13367,"mutex_wait_us":31,"thread_start_us":13202,"threads_started":1}
I20260927 22:45:34.393504 21068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.368879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.1.53.65:40565)
0927 22:45:34.382538 (+ 13659us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.382551 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.385222 (+ 2671us) server_negotiation.cc:407] Connection header received
0927 22:45:34.385522 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.385531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.385758 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.385925 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:34.386879 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.387892 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.390898 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.392407 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.392544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.392778 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.392993 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.393148 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13419,"thread_start_us":13666,"threads_started":1}
I20260927 22:45:34.393687 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.368466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:40565 (local address 127.0.0.1:55996)
0927 22:45:34.385096 (+ 16630us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:34.385116 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:34.385329 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:34.386361 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:34.386365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:34.386385 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:34.386704 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.386711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.388320 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.388326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.390702 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.390718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.392604 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.392611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.392779 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.392821 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:45:34.392940 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:45:34.393074 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16489,"mutex_wait_us":23,"thread_start_us":10978,"threads_started":1}
I20260927 22:45:34.394609 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.368220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.1.53.66:40603)
0927 22:45:34.384791 (+ 16571us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.384805 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.384835 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:45:34.384926 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.384933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.385201 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.385392 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:34.386192 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.387840 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.390262 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.391569 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.391694 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.391882 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.394086 (+ 2204us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.394255 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16360,"thread_start_us":16189,"threads_started":1}
I20260927 22:45:34.396543 20986 tablet_service.cc:1467] Processing CreateTablet for tablet fc336a5702f9448ebfe63d7e1f0e1baf (DEFAULT_TABLE table=test-table [id=8bb1d4fa06f24d41b3f922fb1599149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:34.397370 20718 tablet_service.cc:1467] Processing CreateTablet for tablet fc336a5702f9448ebfe63d7e1f0e1baf (DEFAULT_TABLE table=test-table [id=8bb1d4fa06f24d41b3f922fb1599149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:34.398034 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc336a5702f9448ebfe63d7e1f0e1baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:34.398857 20718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc336a5702f9448ebfe63d7e1f0e1baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:34.398453 20852 tablet_service.cc:1467] Processing CreateTablet for tablet fc336a5702f9448ebfe63d7e1f0e1baf (DEFAULT_TABLE table=test-table [id=8bb1d4fa06f24d41b3f922fb1599149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:34.399775 20852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc336a5702f9448ebfe63d7e1f0e1baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:34.409397 21070 tablet_bootstrap.cc:492] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Bootstrap starting.
I20260927 22:45:34.413347 21070 tablet_bootstrap.cc:654] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:34.413820 21071 tablet_bootstrap.cc:492] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Bootstrap starting.
I20260927 22:45:34.414920 21070 log.cc:824] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:34.415516 21072 tablet_bootstrap.cc:492] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Bootstrap starting.
I20260927 22:45:34.418849 21071 tablet_bootstrap.cc:654] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:34.418995 21072 tablet_bootstrap.cc:654] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:34.419981 21072 log.cc:824] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:34.420428 21071 log.cc:824] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:34.422995 21072 tablet_bootstrap.cc:492] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: No bootstrap required, opened a new log
I20260927 22:45:34.423254 21071 tablet_bootstrap.cc:492] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: No bootstrap required, opened a new log
I20260927 22:45:34.423261 21070 tablet_bootstrap.cc:492] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: No bootstrap required, opened a new log
I20260927 22:45:34.423295 21072 ts_tablet_manager.cc:1397] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:45:34.423524 21071 ts_tablet_manager.cc:1397] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:45:34.423538 21070 ts_tablet_manager.cc:1397] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260927 22:45:34.431560 21072 raft_consensus.cc:348] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.431975 21072 raft_consensus.cc:374] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:34.432068 21072 raft_consensus.cc:729] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a145bb6fff7540509a840dec27a33568, State: Initialized, Role: FOLLOWER
I20260927 22:45:34.432765 21072 consensus_queue.cc:260] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.434780 21071 raft_consensus.cc:348] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.434773 21070 raft_consensus.cc:348] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.435248 21070 raft_consensus.cc:374] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:34.435248 21071 raft_consensus.cc:374] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:34.435362 21071 raft_consensus.cc:729] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc16a6835d6e419d92b52a8bad00a3f1, State: Initialized, Role: FOLLOWER
I20260927 22:45:34.435362 21070 raft_consensus.cc:729] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9b01ee31b94075b5cfbb8a4ac38827, State: Initialized, Role: FOLLOWER
I20260927 22:45:34.436124 20917 heartbeater.cc:503] Master 127.1.53.126:42463 was elected leader, sending a full tablet report...
I20260927 22:45:34.436156 21071 consensus_queue.cc:260] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.436928 21072 ts_tablet_manager.cc:1428] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260927 22:45:34.437988 21071 ts_tablet_manager.cc:1428] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 22:45:34.437806 21070 consensus_queue.cc:260] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.439079 21051 heartbeater.cc:503] Master 127.1.53.126:42463 was elected leader, sending a full tablet report...
I20260927 22:45:34.439743 21070 ts_tablet_manager.cc:1428] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260927 22:45:34.446533 20784 tablet.cc:2369] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:45:34.493033 20918 tablet.cc:2369] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:45:34.505344 21078 raft_consensus.cc:482] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:34.505667 21078 raft_consensus.cc:504] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.507416 21078 leader_election.cc:288] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a145bb6fff7540509a840dec27a33568 (127.1.53.66:40603), cc16a6835d6e419d92b52a8bad00a3f1 (127.1.53.65:40565)
I20260927 22:45:34.513917 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.507811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:46469 (local address 127.1.53.66:40603)
0927 22:45:34.508026 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.508032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.508052 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:45:34.508227 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.508233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.508359 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.508455 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:34.508940 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.509953 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.511907 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.513075 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.513182 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.513254 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.513602 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.513720 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:45:34.514022 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.507677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:40603 (local address 127.1.53.67:46469)
0927 22:45:34.507918 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:34.507939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:34.508067 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:34.508494 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:34.508503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:34.508534 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:34.508801 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.508815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.510126 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.510133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.511733 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.511746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.513233 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.513240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.513430 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.513466 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:45:34.513571 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:45:34.513718 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":1}
I20260927 22:45:34.515044 21068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.508103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:35051 (local address 127.1.53.65:40565)
0927 22:45:34.508386 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.508394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.508839 (+ 445us) server_negotiation.cc:407] Connection header received
0927 22:45:34.509070 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.509074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.509141 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.509223 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:34.509791 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.510811 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.512876 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.514062 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.514209 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.514322 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.514771 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.514876 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:45:34.515240 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.507702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:40565 (local address 127.1.53.67:35051)
0927 22:45:34.508709 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:34.508726 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:34.508862 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:34.509267 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:34.509272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:34.509296 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:34.509639 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.509649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.510959 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.510962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.512707 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:34.512723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.514457 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.514461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:34.514575 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.514635 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 22:45:34.514736 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:45:34.514871 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":185,"thread_start_us":447,"threads_started":1}
I20260927 22:45:34.515158 20872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc336a5702f9448ebfe63d7e1f0e1baf" candidate_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a145bb6fff7540509a840dec27a33568" is_pre_election: true
I20260927 22:45:34.515753 20872 raft_consensus.cc:2454] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9b01ee31b94075b5cfbb8a4ac38827 in term 0.
I20260927 22:45:34.516357 20738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc336a5702f9448ebfe63d7e1f0e1baf" candidate_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" is_pre_election: true
I20260927 22:45:34.516458 20939 leader_election.cc:302] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9b01ee31b94075b5cfbb8a4ac38827, a145bb6fff7540509a840dec27a33568; no voters:
I20260927 22:45:34.516841 21078 raft_consensus.cc:2781] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:45:34.516860 20738 raft_consensus.cc:2454] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9b01ee31b94075b5cfbb8a4ac38827 in term 0.
I20260927 22:45:34.516968 21078 raft_consensus.cc:482] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:45:34.517084 21078 raft_consensus.cc:3037] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:34.519801 21078 raft_consensus.cc:504] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.520572 21078 leader_election.cc:288] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [CANDIDATE]: Term 1 election: Requested vote from peers a145bb6fff7540509a840dec27a33568 (127.1.53.66:40603), cc16a6835d6e419d92b52a8bad00a3f1 (127.1.53.65:40565)
I20260927 22:45:34.520973 20872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc336a5702f9448ebfe63d7e1f0e1baf" candidate_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a145bb6fff7540509a840dec27a33568"
I20260927 22:45:34.521209 20872 raft_consensus.cc:3037] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:34.521606 20738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc336a5702f9448ebfe63d7e1f0e1baf" candidate_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc16a6835d6e419d92b52a8bad00a3f1"
I20260927 22:45:34.521847 20738 raft_consensus.cc:3037] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:34.524067 20738 raft_consensus.cc:2454] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9b01ee31b94075b5cfbb8a4ac38827 in term 1.
I20260927 22:45:34.524461 20872 raft_consensus.cc:2454] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9b01ee31b94075b5cfbb8a4ac38827 in term 1.
I20260927 22:45:34.524582 20942 leader_election.cc:302] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9b01ee31b94075b5cfbb8a4ac38827, cc16a6835d6e419d92b52a8bad00a3f1; no voters:
I20260927 22:45:34.524976 21078 raft_consensus.cc:2781] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:34.525437 21078 raft_consensus.cc:686] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9b01ee31b94075b5cfbb8a4ac38827, State: Running, Role: LEADER
I20260927 22:45:34.526062 21078 consensus_queue.cc:237] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } }
I20260927 22:45:34.531397 20594 catalog_manager.cc:5495] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9b01ee31b94075b5cfbb8a4ac38827 (127.1.53.67). New cstate: current_term: 1 leader_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9b01ee31b94075b5cfbb8a4ac38827" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 } health_report { overall_health: UNKNOWN } } }
W20260927 22:45:34.541702 21052 tablet.cc:2369] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:45:34.563308 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.558202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.1.53.67:37181)
0927 22:45:34.558424 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.558429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.558448 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:45:34.558597 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.558600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.558685 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.558814 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:45:34.559120 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.559857 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.561234 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.561632 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.561736 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.562184 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:45:34.562689 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:45:34.562796 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.562879 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.562947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:45:34.572456 20872 raft_consensus.cc:1261] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 1 FOLLOWER]: Refusing update from remote peer 8c9b01ee31b94075b5cfbb8a4ac38827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:34.572458 20738 raft_consensus.cc:1261] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 1 FOLLOWER]: Refusing update from remote peer 8c9b01ee31b94075b5cfbb8a4ac38827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:34.573477 21078 consensus_queue.cc:1035] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc16a6835d6e419d92b52a8bad00a3f1" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:34.573863 21080 consensus_queue.cc:1035] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a145bb6fff7540509a840dec27a33568" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:34.592602 21085 mvcc.cc:205] Tried to move back new op lower bound from 7334089255195729920 to 7334089255023136768. Current Snapshot: MvccSnapshot[applied={T|T < 7334089255195729920}]
I20260927 22:45:34.602726 20594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:34.603025 20594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:34.606951 20594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d48b5fc02d843738ac6e7373660a862: Sending DeleteTablet for 3 replicas of tablet fc336a5702f9448ebfe63d7e1f0e1baf
I20260927 22:45:34.608270 20986 tablet_service.cc:1514] Processing DeleteTablet for tablet fc336a5702f9448ebfe63d7e1f0e1baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:34 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:56830
I20260927 22:45:34.608279 20852 tablet_service.cc:1514] Processing DeleteTablet for tablet fc336a5702f9448ebfe63d7e1f0e1baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:34 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:45642
I20260927 22:45:34.608357 20718 tablet_service.cc:1514] Processing DeleteTablet for tablet fc336a5702f9448ebfe63d7e1f0e1baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:34 UTC) from {username='kudu', principal='kudu/127.1.53.126@KRBTEST.COM'} at 127.0.0.1:55996
I20260927 22:45:34.612357 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20655
I20260927 22:45:34.620690 21093 tablet_replica.cc:330] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: stopping tablet replica
I20260927 22:45:34.620687 21094 tablet_replica.cc:330] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: stopping tablet replica
I20260927 22:45:34.621223 21093 raft_consensus.cc:2229] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:45:34.621287 21094 raft_consensus.cc:2229] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:34.621955 21094 raft_consensus.cc:2258] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:34.621958 21093 raft_consensus.cc:2258] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:34.622884 21092 tablet_replica.cc:330] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: stopping tablet replica
I20260927 22:45:34.623411 21092 raft_consensus.cc:2229] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:34.624450 21092 raft_consensus.cc:2258] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:34.625051 21093 ts_tablet_manager.cc:1905] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:34.625424 21094 ts_tablet_manager.cc:1905] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:34.626642 21092 ts_tablet_manager.cc:1905] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:34.633122 21094 ts_tablet_manager.cc:1918] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:34.633337 21094 log.cc:1196] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/fc336a5702f9448ebfe63d7e1f0e1baf
I20260927 22:45:34.633347 21093 ts_tablet_manager.cc:1918] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:34.633554 21093 log.cc:1196] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/fc336a5702f9448ebfe63d7e1f0e1baf
I20260927 22:45:34.634038 21094 ts_tablet_manager.cc:1939] T fc336a5702f9448ebfe63d7e1f0e1baf P cc16a6835d6e419d92b52a8bad00a3f1: Deleting consensus metadata
I20260927 22:45:34.634138 21093 ts_tablet_manager.cc:1939] T fc336a5702f9448ebfe63d7e1f0e1baf P 8c9b01ee31b94075b5cfbb8a4ac38827: Deleting consensus metadata
I20260927 22:45:34.636132 20581 catalog_manager.cc:4841] TS cc16a6835d6e419d92b52a8bad00a3f1 (127.1.53.65:40565): tablet fc336a5702f9448ebfe63d7e1f0e1baf (table test-table [id=8bb1d4fa06f24d41b3f922fb1599149e]) successfully deleted
I20260927 22:45:34.636509 20579 catalog_manager.cc:4841] TS 8c9b01ee31b94075b5cfbb8a4ac38827 (127.1.53.67:37181): tablet fc336a5702f9448ebfe63d7e1f0e1baf (table test-table [id=8bb1d4fa06f24d41b3f922fb1599149e]) successfully deleted
I20260927 22:45:34.638475 21092 ts_tablet_manager.cc:1918] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:34.638657 21092 log.cc:1196] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/fc336a5702f9448ebfe63d7e1f0e1baf
I20260927 22:45:34.639245 21092 ts_tablet_manager.cc:1939] T fc336a5702f9448ebfe63d7e1f0e1baf P a145bb6fff7540509a840dec27a33568: Deleting consensus metadata
I20260927 22:45:34.641144 20578 catalog_manager.cc:4841] TS a145bb6fff7540509a840dec27a33568 (127.1.53.66:40603): tablet fc336a5702f9448ebfe63d7e1f0e1baf (table test-table [id=8bb1d4fa06f24d41b3f922fb1599149e]) successfully deleted
Sep 27 22:45:34 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.65@KRBTEST.COM
I20260927 22:45:34.652529 21068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.613960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.1.53.65:40565)
0927 22:45:34.614150 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.614156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.634626 (+ 20470us) server_negotiation.cc:407] Connection header received
0927 22:45:34.634923 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.634928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.635020 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.635100 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:34.637213 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.638352 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.640424 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.641093 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.641234 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.646399 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:34.646416 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:34.646431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:34.646470 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:34.649516 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.650391 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.650398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.650400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.650542 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.650964 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.650967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.650969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.651343 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:34.651569 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.651957 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.652126 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:45:34.653213 20778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:34.745702 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20655
I20260927 22:45:34.764118 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20789
Sep 27 22:45:34 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.66@KRBTEST.COM
I20260927 22:45:34.780838 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.765237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.1.53.66:40603)
0927 22:45:34.765419 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.765425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.765496 (+ 71us) server_negotiation.cc:407] Connection header received
0927 22:45:34.765560 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.765565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.765632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.765784 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:34.767044 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.767823 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.769234 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.769682 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.769820 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.774663 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:34.774679 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:34.774701 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:34.774792 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:34.777881 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.778586 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.778594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.778596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.778721 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.779092 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.779098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.779101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.779562 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:34.779865 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.780193 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.780437 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:45:34.781863 20912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:34.888959 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20789
I20260927 22:45:34.912287 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20923
Sep 27 22:45:34 dist-test-slave-pp80 krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.67@KRBTEST.COM
I20260927 22:45:34.930749 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:34.913116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.1.53.67:37181)
0927 22:45:34.913332 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:34.913338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:34.913360 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:45:34.913532 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:34.913539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:34.913639 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:34.913753 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:34.915075 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.916454 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.917820 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:34.918554 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:34.918696 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:34.923314 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:34.923335 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:34.923363 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:34.923420 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:34.927630 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.928447 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.928457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.928460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.928662 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:34.929026 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:34.929032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:34.929036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:34.929549 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:34.929878 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:34.930088 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:45:34.930326 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:45:34.931631 21046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:35.055264 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20923
I20260927 22:45:35.075464 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 20562
I20260927 22:45:35.076756 20624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:35.189576 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 20562
2026-09-27T22:45:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:35Z Disabled control of system clock
I20260927 22:45:35.269369 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45803
--webserver_interface=127.1.53.126
--webserver_port=0
--builtin_ntp_servers=127.1.53.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:45:35.618909 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:35.619323 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:35.619494 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:35.627825 21103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:35.627996 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:35.628096 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:35.628221 21103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:35.628335 21103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:35.638550 21103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:45803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:35.639935 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:35.642007 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:35.655275 21108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:35.655306 21109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:35.655344 21111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:35.657019 21103 server_base.cc:1034] running on GCE node
I20260927 22:45:35.657914 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:35.659751 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:35.661087 21103 hybrid_clock.cc:648] HybridClock initialized: now 1790549135661039 us; error 60 us; skew 500 ppm
I20260927 22:45:35.661515 21103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:35.663968 21103 webserver.cc:460] Webserver started at http://127.1.53.126:37943/ using document root <none> and password file <none>
I20260927 22:45:35.664849 21103 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:35.664986 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:35.665321 21103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:35.668488 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "c273d5b468ff48afbfb08e62d7de33b9"
format_stamp: "Formatted at 2026-09-27 22:45:35 on dist-test-slave-pp80"
server_key: "e6239cef4c4d95e366699fe0a0139c15"
server_key_iv: "dbb2ab0611625548232ef87cf5de51b8"
server_key_version: "encryptionkey@0"
I20260927 22:45:35.669351 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "c273d5b468ff48afbfb08e62d7de33b9"
format_stamp: "Formatted at 2026-09-27 22:45:35 on dist-test-slave-pp80"
server_key: "e6239cef4c4d95e366699fe0a0139c15"
server_key_iv: "dbb2ab0611625548232ef87cf5de51b8"
server_key_version: "encryptionkey@0"
I20260927 22:45:35.675141 21103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:45:35.679162 21117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:35.681039 21103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:45:35.681273 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "c273d5b468ff48afbfb08e62d7de33b9"
format_stamp: "Formatted at 2026-09-27 22:45:35 on dist-test-slave-pp80"
server_key: "e6239cef4c4d95e366699fe0a0139c15"
server_key_iv: "dbb2ab0611625548232ef87cf5de51b8"
server_key_version: "encryptionkey@0"
I20260927 22:45:35.681561 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:35.708173 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:35.709306 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:35.709652 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:35.730964 21103 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:45803
I20260927 22:45:35.730984 21168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:45803 every 8 connection(s)
I20260927 22:45:35.732594 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:35.735610 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21103
I20260927 22:45:35.735805 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:35.736382 1237 external_mini_cluster.cc:1428] Setting key cc09b6c56667bfc94c43b5ca8a39b63f
I20260927 22:45:35.736907 21169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:35.751205 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9: Bootstrap starting.
I20260927 22:45:35.751875 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:35.738448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.1.53.126:45803)
0927 22:45:35.739539 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:35.739551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:35.739606 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:45:35.741796 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:35.741822 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:35.742427 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:35.742814 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:35.743469 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:35.745085 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:35.746943 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:35.747723 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:35.747878 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:35.749227 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:35.749236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:35.749251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:35.749855 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:35.749944 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:35.749966 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:35.750079 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:35.750312 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:45:35.750695 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
I20260927 22:45:35.754573 21169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:35.755743 21169 log.cc:824] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:35.758980 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9: No bootstrap required, opened a new log
I20260927 22:45:35.766816 21169 raft_consensus.cc:348] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c273d5b468ff48afbfb08e62d7de33b9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45803 } }
I20260927 22:45:35.767267 21169 raft_consensus.cc:374] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:35.767380 21169 raft_consensus.cc:729] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c273d5b468ff48afbfb08e62d7de33b9, State: Initialized, Role: FOLLOWER
I20260927 22:45:35.768038 21169 consensus_queue.cc:260] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c273d5b468ff48afbfb08e62d7de33b9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45803 } }
I20260927 22:45:35.768343 21169 raft_consensus.cc:388] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:35.768498 21169 raft_consensus.cc:482] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:35.768678 21169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:35.771003 21169 raft_consensus.cc:504] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c273d5b468ff48afbfb08e62d7de33b9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45803 } }
I20260927 22:45:35.771631 21169 leader_election.cc:302] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c273d5b468ff48afbfb08e62d7de33b9; no voters:
I20260927 22:45:35.772451 21169 leader_election.cc:288] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:35.772644 21174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:35.773437 21174 raft_consensus.cc:686] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [term 1 LEADER]: Becoming Leader. State: Replica: c273d5b468ff48afbfb08e62d7de33b9, State: Running, Role: LEADER
I20260927 22:45:35.774165 21174 consensus_queue.cc:237] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c273d5b468ff48afbfb08e62d7de33b9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45803 } }
I20260927 22:45:35.774835 21169 sys_catalog.cc:564] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:35.777591 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c273d5b468ff48afbfb08e62d7de33b9. Latest consensus state: current_term: 1 leader_uuid: "c273d5b468ff48afbfb08e62d7de33b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c273d5b468ff48afbfb08e62d7de33b9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45803 } } }
I20260927 22:45:35.778064 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:35.778576 21175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c273d5b468ff48afbfb08e62d7de33b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c273d5b468ff48afbfb08e62d7de33b9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45803 } } }
I20260927 22:45:35.778967 21175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:35.780095 21183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:35.786141 21183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:35.796461 21183 catalog_manager.cc:1347] Generated new cluster ID: dd79dcdb354747fbaacb55e4d5851308
I20260927 22:45:35.796617 21183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:35.805619 21183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:35.807035 21183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:35.814610 21183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9: Generated new TSK 0
I20260927 22:45:35.815599 21183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:35.827231 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.65:0
--local_ip_for_outbound_sockets=127.1.53.65
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45803
--builtin_ntp_servers=127.1.53.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:45:36.188336 21193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:36.188678 21193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:36.188777 21193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:36.196949 21193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:36.197115 21193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:36.197304 21193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.65
I20260927 22:45:36.206914 21193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.53.65
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.65
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:36.208344 21193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:36.210223 21193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:36.224290 21198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:36.224498 21201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:36.224522 21199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:36.225551 21193 server_base.cc:1034] running on GCE node
I20260927 22:45:36.226289 21193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:36.227336 21193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:36.228586 21193 hybrid_clock.cc:648] HybridClock initialized: now 1790549136228537 us; error 60 us; skew 500 ppm
I20260927 22:45:36.228925 21193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:36.231300 21193 webserver.cc:460] Webserver started at http://127.1.53.65:36313/ using document root <none> and password file <none>
I20260927 22:45:36.232079 21193 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:36.232213 21193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:36.232570 21193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:36.235543 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data/instance:
uuid: "3937e4b56cbb47d6a912a5c8f4738c04"
format_stamp: "Formatted at 2026-09-27 22:45:36 on dist-test-slave-pp80"
server_key: "d1fcd18b1ffa1df130a995edee831b07"
server_key_iv: "6b5b3b77299531047cfc4bf7dc6ebb23"
server_key_version: "encryptionkey@0"
I20260927 22:45:36.236452 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance:
uuid: "3937e4b56cbb47d6a912a5c8f4738c04"
format_stamp: "Formatted at 2026-09-27 22:45:36 on dist-test-slave-pp80"
server_key: "d1fcd18b1ffa1df130a995edee831b07"
server_key_iv: "6b5b3b77299531047cfc4bf7dc6ebb23"
server_key_version: "encryptionkey@0"
I20260927 22:45:36.241745 21193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:45:36.245605 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:36.247171 21193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:45:36.247371 21193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal
uuid: "3937e4b56cbb47d6a912a5c8f4738c04"
format_stamp: "Formatted at 2026-09-27 22:45:36 on dist-test-slave-pp80"
server_key: "d1fcd18b1ffa1df130a995edee831b07"
server_key_iv: "6b5b3b77299531047cfc4bf7dc6ebb23"
server_key_version: "encryptionkey@0"
I20260927 22:45:36.247654 21193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:36.272606 21193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:36.273537 21193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:36.273845 21193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:36.275008 21193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:36.276854 21193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:36.276986 21193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:36.277122 21193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:36.277210 21193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:36.311944 21193 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.65:38677
I20260927 22:45:36.312047 21319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.65:38677 every 8 connection(s)
I20260927 22:45:36.313493 21193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/data/info.pb
I20260927 22:45:36.319638 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21193
I20260927 22:45:36.319824 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal/instance
I20260927 22:45:36.320314 1237 external_mini_cluster.cc:1428] Setting key fbd6fba135d037db1a83bfc7c4a9312d
I20260927 22:45:36.323406 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.66:0
--local_ip_for_outbound_sockets=127.1.53.66
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45803
--builtin_ntp_servers=127.1.53.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:45:36.328208 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:36.316446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.65:45043 (local address 127.1.53.126:45803)
0927 22:45:36.316931 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:36.316940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:36.318170 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 22:45:36.320187 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:36.320193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:36.320339 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:36.320469 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:36.321966 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.322986 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.324784 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.325528 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.325683 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:36.327210 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:36.327218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:36.327227 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:36.327255 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:36.327313 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:36.327332 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:36.327429 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:36.327645 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:45:36.327765 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":216,"threads_started":1}
I20260927 22:45:36.329238 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:36.317047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45803 (local address 127.1.53.65:45043)
0927 22:45:36.317951 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:36.317996 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:36.319848 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:36.320764 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:36.320773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:36.320919 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:36.321792 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:36.321817 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.323159 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.323163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:36.324636 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:36.324649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.325760 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.325765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:36.325993 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:36.326813 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:36.326816 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:36.326857 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:36.326948 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:36.327432 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:36.327441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:36.327446 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:45:36.327603 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 22:45:36.327931 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":260,"threads_started":1}
I20260927 22:45:36.330927 21320 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45803
I20260927 22:45:36.331282 21320 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:36.332067 21320 heartbeater.cc:511] Master 127.1.53.126:45803 requested a full tablet report, sending...
I20260927 22:45:36.334267 21134 ts_manager.cc:194] Registered new tserver with Master: 3937e4b56cbb47d6a912a5c8f4738c04 (127.1.53.65:38677)
I20260927 22:45:36.335753 21134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.65:45043
W20260927 22:45:36.669317 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:36.669622 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:36.669778 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:36.677604 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:36.677721 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:36.677891 21324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.66
I20260927 22:45:36.687848 21324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.53.66
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.66
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:36.689229 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:36.691058 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:36.704689 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:36.704665 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:36.704763 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:36.705561 21324 server_base.cc:1034] running on GCE node
I20260927 22:45:36.706262 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:36.707340 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:36.708580 21324 hybrid_clock.cc:648] HybridClock initialized: now 1790549136708503 us; error 79 us; skew 500 ppm
I20260927 22:45:36.708925 21324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:36.711396 21324 webserver.cc:460] Webserver started at http://127.1.53.66:34399/ using document root <none> and password file <none>
I20260927 22:45:36.712138 21324 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:36.712316 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:36.712661 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:36.715826 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data/instance:
uuid: "eda88a370f1547bdb7433616501766e3"
format_stamp: "Formatted at 2026-09-27 22:45:36 on dist-test-slave-pp80"
server_key: "dc3042780f3063f75850f9522fd63309"
server_key_iv: "e4f13ba41cd5d3ef761c9dd5a46b0126"
server_key_version: "encryptionkey@0"
I20260927 22:45:36.716692 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance:
uuid: "eda88a370f1547bdb7433616501766e3"
format_stamp: "Formatted at 2026-09-27 22:45:36 on dist-test-slave-pp80"
server_key: "dc3042780f3063f75850f9522fd63309"
server_key_iv: "e4f13ba41cd5d3ef761c9dd5a46b0126"
server_key_version: "encryptionkey@0"
I20260927 22:45:36.721889 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:45:36.725620 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:36.727056 21324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:36.727257 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal
uuid: "eda88a370f1547bdb7433616501766e3"
format_stamp: "Formatted at 2026-09-27 22:45:36 on dist-test-slave-pp80"
server_key: "dc3042780f3063f75850f9522fd63309"
server_key_iv: "e4f13ba41cd5d3ef761c9dd5a46b0126"
server_key_version: "encryptionkey@0"
I20260927 22:45:36.727491 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:36.742959 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:36.743826 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:36.744431 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:36.745445 21324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:36.747036 21324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:36.747161 21324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:36.747305 21324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:36.747399 21324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:36.781883 21324 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.66:41561
I20260927 22:45:36.781934 21450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.66:41561 every 8 connection(s)
I20260927 22:45:36.783272 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/data/info.pb
I20260927 22:45:36.790472 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21324
I20260927 22:45:36.790696 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal/instance
I20260927 22:45:36.791181 1237 external_mini_cluster.cc:1428] Setting key f61a6852251a49dd727ad37805fc1923
I20260927 22:45:36.794353 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.53.67:0
--local_ip_for_outbound_sockets=127.1.53.67
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45803
--builtin_ntp_servers=127.1.53.84:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:45:36.797484 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:36.786020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:45937 (local address 127.1.53.126:45803)
0927 22:45:36.786277 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:36.786282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:36.787759 (+ 1477us) server_negotiation.cc:407] Connection header received
0927 22:45:36.789747 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:36.789753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:36.789852 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:36.789967 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:36.791224 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.792063 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.794171 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.794879 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.795022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:36.796495 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:36.796504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:36.796512 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:36.796538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:36.796582 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:36.796601 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:36.796704 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:36.796982 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 22:45:36.797098 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 22:45:36.798504 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:36.786569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45803 (local address 127.1.53.66:45937)
0927 22:45:36.787588 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:36.787631 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:36.789452 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:36.790240 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:36.790249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:36.790346 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:36.791046 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:36.791066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.792288 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.792295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:36.794010 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:36.794023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:36.795145 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:36.795151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:36.795366 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:36.796108 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:36.796111 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:36.796156 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:36.796290 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:36.796728 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:36.796738 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:36.796745 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:45:36.796926 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 22:45:36.797331 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":355,"threads_started":1}
I20260927 22:45:36.800169 21451 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45803
I20260927 22:45:36.800563 21451 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:36.801477 21451 heartbeater.cc:511] Master 127.1.53.126:45803 requested a full tablet report, sending...
I20260927 22:45:36.803158 21134 ts_manager.cc:194] Registered new tserver with Master: eda88a370f1547bdb7433616501766e3 (127.1.53.66:41561)
I20260927 22:45:36.804081 21134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.66:45937
W20260927 22:45:37.135842 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:37.136210 21454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:37.136407 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:37.143868 21454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:37.144032 21454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:37.144232 21454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.53.67
I20260927 22:45:37.153853 21454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.53.67
--webserver_port=0
--tserver_master_addrs=127.1.53.126:45803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.53.67
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:37.155207 21454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:37.157150 21454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:37.170639 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:37.170616 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:37.170745 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:37.171362 21454 server_base.cc:1034] running on GCE node
I20260927 22:45:37.172125 21454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:37.173229 21454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:37.174501 21454 hybrid_clock.cc:648] HybridClock initialized: now 1790549137174354 us; error 132 us; skew 500 ppm
I20260927 22:45:37.174877 21454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:45:37.177845 21454 webserver.cc:460] Webserver started at http://127.1.53.67:45729/ using document root <none> and password file <none>
I20260927 22:45:37.178639 21454 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:37.178783 21454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:37.179160 21454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:37.182489 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data/instance:
uuid: "b5b52911a8ff4fbfaf0a35f031c99c88"
format_stamp: "Formatted at 2026-09-27 22:45:37 on dist-test-slave-pp80"
server_key: "a57d306fc200b0383c5a66d771b6ce1c"
server_key_iv: "ecb2f3e9acc9a6ef5b294a98061d735b"
server_key_version: "encryptionkey@0"
I20260927 22:45:37.183353 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance:
uuid: "b5b52911a8ff4fbfaf0a35f031c99c88"
format_stamp: "Formatted at 2026-09-27 22:45:37 on dist-test-slave-pp80"
server_key: "a57d306fc200b0383c5a66d771b6ce1c"
server_key_iv: "ecb2f3e9acc9a6ef5b294a98061d735b"
server_key_version: "encryptionkey@0"
I20260927 22:45:37.189055 21454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:45:37.193159 21468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:37.194669 21454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:37.194859 21454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal
uuid: "b5b52911a8ff4fbfaf0a35f031c99c88"
format_stamp: "Formatted at 2026-09-27 22:45:37 on dist-test-slave-pp80"
server_key: "a57d306fc200b0383c5a66d771b6ce1c"
server_key_iv: "ecb2f3e9acc9a6ef5b294a98061d735b"
server_key_version: "encryptionkey@0"
I20260927 22:45:37.195104 21454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:37.214653 21454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:37.215565 21454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:37.215873 21454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:37.216923 21454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:45:37.218459 21454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:45:37.218597 21454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:37.218743 21454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:45:37.218833 21454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:37.253890 21454 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.67:40945
I20260927 22:45:37.253944 21580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.67:40945 every 8 connection(s)
I20260927 22:45:37.255311 21454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/data/info.pb
I20260927 22:45:37.262290 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21454
I20260927 22:45:37.262506 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal/instance
I20260927 22:45:37.263085 1237 external_mini_cluster.cc:1428] Setting key 8f571a45e82a9a1216704cfd5b9ce436
I20260927 22:45:37.269457 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.258166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.67:36525 (local address 127.1.53.126:45803)
0927 22:45:37.258363 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.258367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.259958 (+ 1591us) server_negotiation.cc:407] Connection header received
0927 22:45:37.262090 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.262096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.262176 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.262273 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:37.263790 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.264865 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.266878 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.267273 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.267377 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.268769 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:37.268774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:37.268777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:37.268796 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:37.268826 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:37.268837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:37.268908 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.269122 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.269206 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:45:37.270551 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.258747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:45803 (local address 127.1.53.67:36525)
0927 22:45:37.259776 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.259820 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:37.261696 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:37.262592 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:37.262604 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:37.262730 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:45:37.263596 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.263617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.265052 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.265060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.266724 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.266737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.267475 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.267480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.267696 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.268456 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:45:37.268459 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:45:37.268500 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:45:37.268593 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:45:37.268922 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:45:37.268929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:45:37.268936 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:45:37.269095 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:45:37.269419 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":347,"threads_started":1}
I20260927 22:45:37.271955 21581 heartbeater.cc:347] Connected to a master server at 127.1.53.126:45803
I20260927 22:45:37.272338 21581 heartbeater.cc:464] Registering TS with master...
I20260927 22:45:37.273165 21581 heartbeater.cc:511] Master 127.1.53.126:45803 requested a full tablet report, sending...
I20260927 22:45:37.274827 21134 ts_manager.cc:194] Registered new tserver with Master: b5b52911a8ff4fbfaf0a35f031c99c88 (127.1.53.67:40945)
I20260927 22:45:37.275754 21134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.53.67:36525
I20260927 22:45:37.278637 1237 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:45:37.288836 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.282879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.1.53.126:45803)
0927 22:45:37.283100 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.283105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.283553 (+ 448us) server_negotiation.cc:407] Connection header received
0927 22:45:37.283665 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.283669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.283736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.283831 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:37.284449 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.285302 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.286715 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.287138 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.287240 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.288182 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:37.288187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:37.288190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:37.288210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:37.288301 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:37.288322 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:37.288393 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.288512 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.288598 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:45:37.292848 21134 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:45:37.300064 21134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:45:37.331779 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.313756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.1.53.67:40945)
0927 22:45:37.321884 (+ 8128us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.321893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.322757 (+ 864us) server_negotiation.cc:407] Connection header received
0927 22:45:37.322893 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.322905 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.323260 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.323499 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:37.325041 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.326589 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.328700 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.330587 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.330721 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.331012 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.331210 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.331386 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7955,"thread_start_us":7805,"threads_started":1}
I20260927 22:45:37.331764 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.313137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:40945 (local address 127.0.0.1:42384)
0927 22:45:37.322418 (+ 9281us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.322442 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:37.322600 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:37.323542 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:37.323552 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:37.323592 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:37.324047 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.324060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.326718 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.326726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.328433 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.328452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.330745 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.330751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.330950 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.331010 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 22:45:37.331164 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:45:37.331410 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9087,"thread_start_us":9158,"threads_started":1}
I20260927 22:45:37.333375 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.312896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47408)
0927 22:45:37.320961 (+ 8065us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.321024 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:37.321236 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:37.325417 (+ 4181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:37.325422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:37.325446 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:37.325835 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.325852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.328057 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.328062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.330453 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.330469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.332439 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.332444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.332725 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.332812 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 22:45:37.332890 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 22:45:37.333021 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7806,"mutex_wait_us":100,"thread_start_us":7655,"threads_started":1}
I20260927 22:45:37.333717 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.313575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.1.53.65:38677)
0927 22:45:37.324792 (+ 11217us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.324802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.324835 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:45:37.324935 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.324943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.325212 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.326079 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:37.326294 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.327905 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.330658 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.332191 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.332831 (+ 640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.333096 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.333163 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.333326 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11046,"thread_start_us":10939,"threads_started":1}
I20260927 22:45:37.334244 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.313522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.66:41561 (local address 127.0.0.1:43304)
0927 22:45:37.325093 (+ 11571us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.325111 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:37.325228 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:37.325926 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:37.325931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:37.325968 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:37.326305 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.326316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.328239 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.328244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.330162 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.330174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.332386 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.332395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.332730 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.333696 (+ 966us) client_negotiation.cc:769] Sending connection context
0927 22:45:37.333793 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:45:37.333929 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11458,"mutex_wait_us":126,"thread_start_us":11307,"threads_started":1}
I20260927 22:45:37.335206 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.314029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.1.53.66:41561)
0927 22:45:37.324336 (+ 10307us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.324349 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.325359 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 22:45:37.325464 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.325473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.325714 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.325964 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:37.326475 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.328036 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.330353 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.331852 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.331980 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.332212 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.334663 (+ 2451us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.334846 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10073,"thread_start_us":9917,"threads_started":1}
I20260927 22:45:37.336354 21516 tablet_service.cc:1467] Processing CreateTablet for tablet a5894f41312e411e860349f72a7892b0 (DEFAULT_TABLE table=test-table [id=d8df1b9224eb4e0f88755ed44c90a082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:37.337190 21255 tablet_service.cc:1467] Processing CreateTablet for tablet a5894f41312e411e860349f72a7892b0 (DEFAULT_TABLE table=test-table [id=d8df1b9224eb4e0f88755ed44c90a082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:37.337966 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5894f41312e411e860349f72a7892b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:37.338340 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5894f41312e411e860349f72a7892b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:37.338701 21386 tablet_service.cc:1467] Processing CreateTablet for tablet a5894f41312e411e860349f72a7892b0 (DEFAULT_TABLE table=test-table [id=d8df1b9224eb4e0f88755ed44c90a082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:45:37.340000 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5894f41312e411e860349f72a7892b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:37.344480 21320 heartbeater.cc:503] Master 127.1.53.126:45803 was elected leader, sending a full tablet report...
I20260927 22:45:37.350839 21599 tablet_bootstrap.cc:492] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Bootstrap starting.
I20260927 22:45:37.354864 21599 tablet_bootstrap.cc:654] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:37.355788 21600 tablet_bootstrap.cc:492] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Bootstrap starting.
I20260927 22:45:37.356191 21599 log.cc:824] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:37.358867 21602 tablet_bootstrap.cc:492] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Bootstrap starting.
I20260927 22:45:37.359349 21600 tablet_bootstrap.cc:654] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:37.359630 21599 tablet_bootstrap.cc:492] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: No bootstrap required, opened a new log
I20260927 22:45:37.359916 21599 ts_tablet_manager.cc:1397] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:45:37.360646 21600 log.cc:824] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:37.362099 21602 tablet_bootstrap.cc:654] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:37.363240 21602 log.cc:824] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:37.363879 21600 tablet_bootstrap.cc:492] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: No bootstrap required, opened a new log
I20260927 22:45:37.364156 21600 ts_tablet_manager.cc:1397] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 22:45:37.366394 21602 tablet_bootstrap.cc:492] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: No bootstrap required, opened a new log
I20260927 22:45:37.366659 21602 ts_tablet_manager.cc:1397] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:45:37.370893 21599 raft_consensus.cc:348] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.371522 21599 raft_consensus.cc:374] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:37.371668 21599 raft_consensus.cc:729] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b52911a8ff4fbfaf0a35f031c99c88, State: Initialized, Role: FOLLOWER
I20260927 22:45:37.372439 21599 consensus_queue.cc:260] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.373652 21581 heartbeater.cc:503] Master 127.1.53.126:45803 was elected leader, sending a full tablet report...
I20260927 22:45:37.374315 21599 ts_tablet_manager.cc:1428] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 22:45:37.374841 21600 raft_consensus.cc:348] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.374904 21602 raft_consensus.cc:348] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.375339 21602 raft_consensus.cc:374] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:37.375334 21600 raft_consensus.cc:374] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:37.375461 21600 raft_consensus.cc:729] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3937e4b56cbb47d6a912a5c8f4738c04, State: Initialized, Role: FOLLOWER
I20260927 22:45:37.375499 21602 raft_consensus.cc:729] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda88a370f1547bdb7433616501766e3, State: Initialized, Role: FOLLOWER
I20260927 22:45:37.376149 21600 consensus_queue.cc:260] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.376169 21602 consensus_queue.cc:260] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.378051 21600 ts_tablet_manager.cc:1428] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 22:45:37.379284 21451 heartbeater.cc:503] Master 127.1.53.126:45803 was elected leader, sending a full tablet report...
I20260927 22:45:37.380617 21602 ts_tablet_manager.cc:1428] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 22:45:37.396420 21606 raft_consensus.cc:482] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:45:37.396736 21606 raft_consensus.cc:504] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.398488 21606 leader_election.cc:288] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3937e4b56cbb47d6a912a5c8f4738c04 (127.1.53.65:38677), b5b52911a8ff4fbfaf0a35f031c99c88 (127.1.53.67:40945)
I20260927 22:45:37.408695 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.398909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:33717 (local address 127.1.53.65:38677)
0927 22:45:37.399130 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.399139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.399972 (+ 833us) server_negotiation.cc:407] Connection header received
0927 22:45:37.400300 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.400305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.400404 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.400491 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:37.401301 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.402369 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.406143 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.407421 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.407589 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.407676 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.408283 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.408436 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:45:37.408694 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.398965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.53.66:35083 (local address 127.1.53.67:40945)
0927 22:45:37.399205 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.399213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.399972 (+ 759us) server_negotiation.cc:407] Connection header received
0927 22:45:37.400213 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.400219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.400333 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.400428 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:45:37.401301 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.402382 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.406162 (+ 3780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.407421 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.407564 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.407674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.408283 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.408429 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:45:37.408812 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.398981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.67:40945 (local address 127.1.53.66:35083)
0927 22:45:37.399867 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.399883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:37.400051 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:37.400450 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:37.400472 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:37.400501 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:37.401085 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.401105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.402572 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.402580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.405831 (+ 3251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.405851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.407756 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.407761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.408005 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.408085 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 22:45:37.408191 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:45:37.408378 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":442,"thread_start_us":456,"threads_started":1}
I20260927 22:45:37.408840 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.398845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.1.53.66:33717)
0927 22:45:37.399862 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.399881 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:45:37.400063 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:45:37.400507 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:45:37.400511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:45:37.400529 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:45:37.401151 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.401160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.402579 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.402584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.405831 (+ 3247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:45:37.405851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.407750 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.407758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:45:37.408005 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.408085 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 22:45:37.408216 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 22:45:37.408378 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":354,"thread_start_us":679,"threads_started":1}
I20260927 22:45:37.410030 21536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894f41312e411e860349f72a7892b0" candidate_uuid: "eda88a370f1547bdb7433616501766e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" is_pre_election: true
I20260927 22:45:37.410127 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894f41312e411e860349f72a7892b0" candidate_uuid: "eda88a370f1547bdb7433616501766e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" is_pre_election: true
I20260927 22:45:37.410681 21275 raft_consensus.cc:2454] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eda88a370f1547bdb7433616501766e3 in term 0.
I20260927 22:45:37.410681 21536 raft_consensus.cc:2454] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eda88a370f1547bdb7433616501766e3 in term 0.
I20260927 22:45:37.411474 21341 leader_election.cc:302] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3937e4b56cbb47d6a912a5c8f4738c04, eda88a370f1547bdb7433616501766e3; no voters:
I20260927 22:45:37.411911 21606 raft_consensus.cc:2781] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:45:37.412083 21606 raft_consensus.cc:482] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:45:37.412186 21606 raft_consensus.cc:3037] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:37.414453 21606 raft_consensus.cc:504] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.415058 21606 leader_election.cc:288] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [CANDIDATE]: Term 1 election: Requested vote from peers 3937e4b56cbb47d6a912a5c8f4738c04 (127.1.53.65:38677), b5b52911a8ff4fbfaf0a35f031c99c88 (127.1.53.67:40945)
I20260927 22:45:37.415637 21536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894f41312e411e860349f72a7892b0" candidate_uuid: "eda88a370f1547bdb7433616501766e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88"
I20260927 22:45:37.415637 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5894f41312e411e860349f72a7892b0" candidate_uuid: "eda88a370f1547bdb7433616501766e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3937e4b56cbb47d6a912a5c8f4738c04"
I20260927 22:45:37.415859 21275 raft_consensus.cc:3037] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:37.415859 21536 raft_consensus.cc:3037] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:37.419305 21536 raft_consensus.cc:2454] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda88a370f1547bdb7433616501766e3 in term 1.
I20260927 22:45:37.419305 21275 raft_consensus.cc:2454] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda88a370f1547bdb7433616501766e3 in term 1.
I20260927 22:45:37.419864 21339 leader_election.cc:302] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3937e4b56cbb47d6a912a5c8f4738c04, b5b52911a8ff4fbfaf0a35f031c99c88, eda88a370f1547bdb7433616501766e3; no voters:
I20260927 22:45:37.420293 21606 raft_consensus.cc:2781] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:37.420742 21606 raft_consensus.cc:686] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 1 LEADER]: Becoming Leader. State: Replica: eda88a370f1547bdb7433616501766e3, State: Running, Role: LEADER
I20260927 22:45:37.421238 21606 consensus_queue.cc:237] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } }
I20260927 22:45:37.426397 21134 catalog_manager.cc:5495] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eda88a370f1547bdb7433616501766e3 (127.1.53.66). New cstate: current_term: 1 leader_uuid: "eda88a370f1547bdb7433616501766e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eda88a370f1547bdb7433616501766e3" member_type: VOTER last_known_addr { host: "127.1.53.66" port: 41561 } health_report { overall_health: HEALTHY } } }
I20260927 22:45:37.443077 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.437688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.1.53.66:41561)
0927 22:45:37.437897 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.437903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.437924 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:45:37.438144 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.438163 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.438239 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.438394 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:45:37.438756 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.439448 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.440912 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.441332 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.441459 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.441980 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:45:37.442498 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:45:37.442670 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.442727 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.442805 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:45:37.451364 21536 raft_consensus.cc:1261] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 1 FOLLOWER]: Refusing update from remote peer eda88a370f1547bdb7433616501766e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:37.451553 21275 raft_consensus.cc:1261] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 1 FOLLOWER]: Refusing update from remote peer eda88a370f1547bdb7433616501766e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:45:37.452606 21610 consensus_queue.cc:1035] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3937e4b56cbb47d6a912a5c8f4738c04" member_type: VOTER last_known_addr { host: "127.1.53.65" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:37.452978 21606 consensus_queue.cc:1035] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b52911a8ff4fbfaf0a35f031c99c88" member_type: VOTER last_known_addr { host: "127.1.53.67" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:45:37.481825 21134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:37.482118 21134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:45:37.486280 21134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c273d5b468ff48afbfb08e62d7de33b9: Sending DeleteTablet for 3 replicas of tablet a5894f41312e411e860349f72a7892b0
I20260927 22:45:37.487571 21255 tablet_service.cc:1514] Processing DeleteTablet for tablet a5894f41312e411e860349f72a7892b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:37 UTC) from {username='slave'} at 127.0.0.1:47408
I20260927 22:45:37.487859 21516 tablet_service.cc:1514] Processing DeleteTablet for tablet a5894f41312e411e860349f72a7892b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:37 UTC) from {username='slave'} at 127.0.0.1:42384
I20260927 22:45:37.487887 21386 tablet_service.cc:1514] Processing DeleteTablet for tablet a5894f41312e411e860349f72a7892b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:45:37 UTC) from {username='slave'} at 127.0.0.1:43304
I20260927 22:45:37.489974 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21193
I20260927 22:45:37.500898 21625 tablet_replica.cc:330] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: stopping tablet replica
I20260927 22:45:37.500947 21623 tablet_replica.cc:330] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: stopping tablet replica
I20260927 22:45:37.501572 21625 raft_consensus.cc:2229] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:45:37.501753 21623 raft_consensus.cc:2229] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:37.502476 21625 raft_consensus.cc:2258] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:37.502524 21623 raft_consensus.cc:2258] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:37.505354 21625 ts_tablet_manager.cc:1905] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:37.507269 21623 ts_tablet_manager.cc:1905] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:37.512416 21624 tablet_replica.cc:330] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: stopping tablet replica
I20260927 22:45:37.512912 21624 raft_consensus.cc:2229] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:45:37.513441 21624 raft_consensus.cc:2258] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:45:37.515890 21624 ts_tablet_manager.cc:1905] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:45:37.516073 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.491741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.1.53.65:38677)
0927 22:45:37.491978 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.491987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.505779 (+ 13792us) server_negotiation.cc:407] Connection header received
0927 22:45:37.506324 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.506330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.506450 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.506530 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:37.507234 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.508357 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.510264 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.510937 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.511042 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.514171 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:37.514183 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:37.514201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:37.514982 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:37.515127 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:37.515159 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:37.515282 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.515511 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.515630 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:45:37.518047 21315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:37.523190 21624 ts_tablet_manager.cc:1918] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:37.523383 21624 log.cc:1196] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-2/wal/wals/a5894f41312e411e860349f72a7892b0
I20260927 22:45:37.523931 21624 ts_tablet_manager.cc:1939] T a5894f41312e411e860349f72a7892b0 P b5b52911a8ff4fbfaf0a35f031c99c88: Deleting consensus metadata
I20260927 22:45:37.525403 21625 ts_tablet_manager.cc:1918] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:37.525682 21625 log.cc:1196] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-1/wal/wals/a5894f41312e411e860349f72a7892b0
I20260927 22:45:37.526065 21118 catalog_manager.cc:4841] TS b5b52911a8ff4fbfaf0a35f031c99c88 (127.1.53.67:40945): tablet a5894f41312e411e860349f72a7892b0 (table test-table [id=d8df1b9224eb4e0f88755ed44c90a082]) successfully deleted
I20260927 22:45:37.526396 21625 ts_tablet_manager.cc:1939] T a5894f41312e411e860349f72a7892b0 P eda88a370f1547bdb7433616501766e3: Deleting consensus metadata
I20260927 22:45:37.528471 21121 catalog_manager.cc:4841] TS eda88a370f1547bdb7433616501766e3 (127.1.53.66:41561): tablet a5894f41312e411e860349f72a7892b0 (table test-table [id=d8df1b9224eb4e0f88755ed44c90a082]) successfully deleted
I20260927 22:45:37.648509 21623 ts_tablet_manager.cc:1918] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:45:37.648829 21623 log.cc:1196] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Deleting WAL directory at /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790548812741971-1237-0/minicluster-data/ts-0/wal/wals/a5894f41312e411e860349f72a7892b0
I20260927 22:45:37.649495 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21193
I20260927 22:45:37.649550 21623 ts_tablet_manager.cc:1939] T a5894f41312e411e860349f72a7892b0 P 3937e4b56cbb47d6a912a5c8f4738c04: Deleting consensus metadata
W20260927 22:45:37.668028 21120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.53.65:38677 (error 108)
I20260927 22:45:37.668777 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21324
I20260927 22:45:37.670310 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.669874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47422)
0927 22:45:37.670048 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.670167 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:45:37.670542 21120 catalog_manager.cc:4584] TS 3937e4b56cbb47d6a912a5c8f4738c04 (127.1.53.65:38677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5894f41312e411e860349f72a7892b0: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
I20260927 22:45:37.677956 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.669616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.1.53.66:41561)
0927 22:45:37.669800 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.669807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.669827 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:45:37.670565 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.670572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.670693 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.670798 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:37.671321 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.672119 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.674090 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.674725 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.674876 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.676280 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:37.676297 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:37.676321 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:37.677039 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:37.677087 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:37.677108 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:37.677233 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.677353 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.677472 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:45:37.678755 21446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:37.724462 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.723866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47436)
0927 22:45:37.724137 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.724228 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:45:37.725533 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.725173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47442)
0927 22:45:37.725330 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.725409 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:45:37.786265 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.785695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47454)
0927 22:45:37.785911 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.786054 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 22:45:37.787361 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.786988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47466)
0927 22:45:37.787167 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.787240 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:45:37.796516 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21324
I20260927 22:45:37.816377 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21454
I20260927 22:45:37.823354 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.817208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.1.53.67:40945)
0927 22:45:37.817414 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:37.817419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:37.817438 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:45:37.817518 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:37.817522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:37.817602 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:37.817706 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:37.818085 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.818860 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.820092 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:37.820699 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:37.820798 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:37.821986 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:37.821994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:37.822009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:37.822589 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:37.822637 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:37.822673 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:37.822756 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:37.822913 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:45:37.823008 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:45:37.824040 21576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:37.868619 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.868023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47472)
0927 22:45:37.868315 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.868428 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 22:45:37.869799 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:37.869350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47474)
0927 22:45:37.869569 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:37.869659 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 22:45:37.915547 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21454
I20260927 22:45:37.933610 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21103
I20260927 22:45:37.934725 21164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:38.045023 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:38.044400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.65:38677 (local address 127.0.0.1:47488)
0927 22:45:38.044658 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 22:45:38.044768 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.65:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:45:38.045275 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21103
2026-09-27T22:45:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2855 ms)
[----------] 6 tests from EncryptionPolicyTest (22715 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:38 dist-test-slave-pp80 krb5kdc[21634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:38 dist-test-slave-pp80 krb5kdc[21634](info): set up 2 sockets
Sep 27 22:45:38 dist-test-slave-pp80 krb5kdc[21634](info): commencing operation
krb5kdc: starting...
W20260927 22:45:39.200730 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:39 dist-test-slave-pp80 krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:39.468863 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:39.810551 21647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:39.810844 21647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:39.811024 21647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:39.818957 21647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:39.819089 21647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:39.819196 21647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:39.819353 21647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:39.819428 21647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:39.829555 21647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:39.830910 21647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:39.832351 21647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:39.844601 21655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:39.844738 21652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:39.844738 21653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:39.845760 21647 server_base.cc:1034] running on GCE node
I20260927 22:45:39.846583 21647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:39.848070 21647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:39.849316 21647 hybrid_clock.cc:648] HybridClock initialized: now 1790549139849271 us; error 61 us; skew 500 ppm
I20260927 22:45:39.849704 21647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:39 dist-test-slave-pp80 krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:39.853036 21647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:39.854569 21647 webserver.cc:460] Webserver started at http://127.0.0.1:42063/ using document root <none> and password file <none>
I20260927 22:45:39.855301 21647 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:39.855456 21647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:39.855790 21647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:39.859253 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "6dea433c5eb748ad9657dc0b6ae38e98"
format_stamp: "Formatted at 2026-09-27 22:45:39 on dist-test-slave-pp80"
server_key: "d0d38ed3d2511d7d78ff8757a373bbd7"
server_key_iv: "7f682fe238605c25a29cc7ef084611b3"
server_key_version: "encryptionkey@0"
I20260927 22:45:39.860226 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "6dea433c5eb748ad9657dc0b6ae38e98"
format_stamp: "Formatted at 2026-09-27 22:45:39 on dist-test-slave-pp80"
server_key: "d0d38ed3d2511d7d78ff8757a373bbd7"
server_key_iv: "7f682fe238605c25a29cc7ef084611b3"
server_key_version: "encryptionkey@0"
I20260927 22:45:39.865635 21647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:45:39.869161 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:39.870702 21647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:39.870850 21647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "6dea433c5eb748ad9657dc0b6ae38e98"
format_stamp: "Formatted at 2026-09-27 22:45:39 on dist-test-slave-pp80"
server_key: "d0d38ed3d2511d7d78ff8757a373bbd7"
server_key_iv: "7f682fe238605c25a29cc7ef084611b3"
server_key_version: "encryptionkey@0"
I20260927 22:45:39.871037 21647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:39.899565 21647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:39.904700 21647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:39.904965 21647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:39.924487 21647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45597
I20260927 22:45:39.924489 21713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45597 every 8 connection(s)
I20260927 22:45:39.926045 21647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:39.929952 21714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:39.935317 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21647
I20260927 22:45:39.935532 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:39.936198 1237 external_mini_cluster.cc:1428] Setting key faf9a4f9f87b375752d5ad7d895991fd
I20260927 22:45:39.941517 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98: Bootstrap starting.
I20260927 22:45:39.945483 21714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:39.946818 21714 log.cc:824] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:39.950330 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98: No bootstrap required, opened a new log
Sep 27 22:45:39 dist-test-slave-pp80 krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:45:39.958674 21714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45597 } }
I20260927 22:45:39.959095 21714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:39.959235 21714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dea433c5eb748ad9657dc0b6ae38e98, State: Initialized, Role: FOLLOWER
I20260927 22:45:39.959909 21714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45597 } }
I20260927 22:45:39.960170 21714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:39.960397 21714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:39.960604 21714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:39.962491 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:39.938477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43171 (local address 127.0.0.1:45597)
0927 22:45:39.939365 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:39.939378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:39.939426 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:45:39.941348 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:39.941373 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:39.941868 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:39.942222 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:39.944372 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:39.945861 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:39.947777 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:39.948710 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:39.948849 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:39.953839 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:39.953852 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:39.953870 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:39.953916 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:39.957376 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:39.958259 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:39.958266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:39.958274 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:39.958446 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:39.958919 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:39.958930 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:39.958932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:39.959658 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:39.960009 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:39.960623 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 22:45:39.961012 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":429,"threads_started":1}
I20260927 22:45:39.963985 21714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45597 } }
I20260927 22:45:39.964623 21714 leader_election.cc:302] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dea433c5eb748ad9657dc0b6ae38e98; no voters:
I20260927 22:45:39.965291 21714 leader_election.cc:288] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:39.965425 21719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:39.967296 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:39.967227 21719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [term 1 LEADER]: Becoming Leader. State: Replica: 6dea433c5eb748ad9657dc0b6ae38e98, State: Running, Role: LEADER
I20260927 22:45:39.967478 21714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:39.967795 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45597 } }
I20260927 22:45:39.977735 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45597 } } }
I20260927 22:45:39.978217 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:39.978745 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dea433c5eb748ad9657dc0b6ae38e98. Latest consensus state: current_term: 1 leader_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dea433c5eb748ad9657dc0b6ae38e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45597 } } }
I20260927 22:45:39.978982 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:39.982383 21734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:39.988844 21734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:39.993971 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:39.973970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60219 (local address 127.0.0.1:45597)
0927 22:45:39.974231 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:39.974240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:39.974885 (+ 645us) server_negotiation.cc:407] Connection header received
0927 22:45:39.975271 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:39.975280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:39.975385 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:39.975476 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:39.977979 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:39.979251 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:39.981416 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:39.982375 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:39.982550 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:39.986113 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:39.986125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:39.986129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:39.986176 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:39.990206 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:39.991175 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:39.991181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:39.991185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:39.991328 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:39.991919 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:39.991925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:39.991930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:39.992410 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:39.992705 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:39.993401 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 22:45:39.993589 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 22:45:40.001186 21734 catalog_manager.cc:1347] Generated new cluster ID: 27e046029bf54dc4a2f4ed9761410e90
I20260927 22:45:40.001365 21734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:40.023885 21734 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:40.025839 21734 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:40.044909 21734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dea433c5eb748ad9657dc0b6ae38e98: Generated new TSK 0
I20260927 22:45:40.046080 21734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:40.057354 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21647
I20260927 22:45:40.062112 21709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:40.159693 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21647
W20260927 22:45:40.177507 21632 connection.cc:535] client connection to 127.0.0.1:45597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45597): Connection reset by peer (error 104)
2026-09-27T22:45:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2108 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:40 dist-test-slave-pp80 krb5kdc[21746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:40 dist-test-slave-pp80 krb5kdc[21746](info): set up 2 sockets
Sep 27 22:45:40 dist-test-slave-pp80 krb5kdc[21746](info): commencing operation
krb5kdc: starting...
W20260927 22:45:41.312661 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:41 dist-test-slave-pp80 krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:41.576996 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:41.954936 21759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:41.955251 21759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:41.955421 21759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:41.962826 21759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:41.962950 21759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:41.963052 21759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:41.963140 21759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:41.963230 21759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:41.973989 21759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:41.975461 21759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:41.976827 21759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:41.989136 21767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:41.989246 21765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:41.989115 21764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:41.990123 21759 server_base.cc:1034] running on GCE node
I20260927 22:45:41.990890 21759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:41.992403 21759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:41.993610 21759 hybrid_clock.cc:648] HybridClock initialized: now 1790549141993564 us; error 43 us; skew 500 ppm
I20260927 22:45:41.993974 21759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:41 dist-test-slave-pp80 krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:41.997375 21759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:41.999022 21759 webserver.cc:460] Webserver started at http://127.0.0.1:38297/ using document root <none> and password file <none>
I20260927 22:45:41.999748 21759 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:41.999855 21759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:42.000203 21759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:42.003306 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "a1f9cead300e4bff9922858a099a07e6"
format_stamp: "Formatted at 2026-09-27 22:45:41 on dist-test-slave-pp80"
server_key: "442df83dd788cd49d1ce17874f017c2a"
server_key_iv: "02b46a261dfe4eb5ba8a03e1412b1ab3"
server_key_version: "encryptionkey@0"
I20260927 22:45:42.004125 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "a1f9cead300e4bff9922858a099a07e6"
format_stamp: "Formatted at 2026-09-27 22:45:41 on dist-test-slave-pp80"
server_key: "442df83dd788cd49d1ce17874f017c2a"
server_key_iv: "02b46a261dfe4eb5ba8a03e1412b1ab3"
server_key_version: "encryptionkey@0"
I20260927 22:45:42.009728 21759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:45:42.013592 21774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:42.015190 21759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:42.015400 21759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "a1f9cead300e4bff9922858a099a07e6"
format_stamp: "Formatted at 2026-09-27 22:45:41 on dist-test-slave-pp80"
server_key: "442df83dd788cd49d1ce17874f017c2a"
server_key_iv: "02b46a261dfe4eb5ba8a03e1412b1ab3"
server_key_version: "encryptionkey@0"
I20260927 22:45:42.015615 21759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:42.032877 21759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:42.037459 21759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:42.037773 21759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:42.057518 21759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40607
I20260927 22:45:42.057574 21825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40607 every 8 connection(s)
I20260927 22:45:42.059018 21759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:42.062853 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:42.064453 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21759
I20260927 22:45:42.064666 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:42.065210 1237 external_mini_cluster.cc:1428] Setting key 6e07d217fda2e763fbe43dad652b5600
I20260927 22:45:42.077623 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6: Bootstrap starting.
I20260927 22:45:42.078949 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:42.067067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59317 (local address 127.0.0.1:40607)
0927 22:45:42.067871 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:42.067882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:42.067923 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:45:42.069579 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:42.069607 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:42.070037 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:42.070393 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:42.070976 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:42.072386 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:42.074280 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:42.075042 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:42.075214 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:42.076567 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:42.076575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:42.076590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:42.077146 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:42.077235 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:42.077264 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:42.077383 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:42.077705 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 22:45:42.077900 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":371,"threads_started":1}
I20260927 22:45:42.081410 21826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:42.082684 21826 log.cc:824] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:42.083212 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:42.085644 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6: No bootstrap required, opened a new log
I20260927 22:45:42.094617 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:42.087760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35309 (local address 127.0.0.1:40607)
0927 22:45:42.087979 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:42.087985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:42.088331 (+ 346us) server_negotiation.cc:407] Connection header received
0927 22:45:42.088582 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:42.088588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:42.088678 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:42.088782 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:42.089319 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:42.090497 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:42.091792 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:42.092535 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:42.092672 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:42.093809 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:42.093818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:42.093828 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:42.093852 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:42.093899 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:42.093919 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:42.094062 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:42.094213 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 22:45:42.094271 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:45:42.095702 21826 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9cead300e4bff9922858a099a07e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 22:45:42.096154 21826 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:42.096297 21826 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f9cead300e4bff9922858a099a07e6, State: Initialized, Role: FOLLOWER
I20260927 22:45:42.096911 21826 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9cead300e4bff9922858a099a07e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 22:45:42.097195 21826 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:42.097345 21826 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:42.097540 21826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:42.099787 21826 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9cead300e4bff9922858a099a07e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 22:45:42.100397 21826 leader_election.cc:302] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f9cead300e4bff9922858a099a07e6; no voters:
I20260927 22:45:42.101042 21826 leader_election.cc:288] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:42.101253 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:42.102387 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [term 1 LEADER]: Becoming Leader. State: Replica: a1f9cead300e4bff9922858a099a07e6, State: Running, Role: LEADER
I20260927 22:45:42.102914 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9cead300e4bff9922858a099a07e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260927 22:45:42.103386 21826 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:42.105741 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f9cead300e4bff9922858a099a07e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9cead300e4bff9922858a099a07e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } } }
I20260927 22:45:42.106390 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:42.106539 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f9cead300e4bff9922858a099a07e6. Latest consensus state: current_term: 1 leader_uuid: "a1f9cead300e4bff9922858a099a07e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9cead300e4bff9922858a099a07e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } } }
I20260927 22:45:42.106925 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:42.107393 21842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:42.114279 21842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:42.125766 21842 catalog_manager.cc:1347] Generated new cluster ID: d87ff99f0b984117b85ec3a1cb1c800f
I20260927 22:45:42.125897 21842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:42.140213 21842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:42.141577 21842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:42.153456 21842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f9cead300e4bff9922858a099a07e6: Generated new TSK 0
I20260927 22:45:42.154369 21842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:42.162734 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21759
W20260927 22:45:42.164151 21821 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59317
W20260927 22:45:42.164922 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21759: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:45:42.165117 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21759
W20260927 22:45:42.182533 21744 connection.cc:535] client connection to 127.0.0.1:40607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40607): Connection reset by peer (error 104)
2026-09-27T22:45:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:42 dist-test-slave-pp80 krb5kdc[21857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:42 dist-test-slave-pp80 krb5kdc[21857](info): set up 2 sockets
Sep 27 22:45:42 dist-test-slave-pp80 krb5kdc[21857](info): commencing operation
krb5kdc: starting...
W20260927 22:45:43.293208 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:43 dist-test-slave-pp80 krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:43.559887 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:43.909484 21870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:43.909883 21870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:43.910066 21870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:43.917663 21870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:43.917868 21870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:43.917989 21870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:43.918087 21870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:43.918164 21870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:43.928484 21870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:43.929919 21870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:43.931308 21870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:43.943984 21875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:43.944056 21876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:43.944052 21878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:43.949877 21870 server_base.cc:1034] running on GCE node
I20260927 22:45:43.950764 21870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:43.952351 21870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:43.953577 21870 hybrid_clock.cc:648] HybridClock initialized: now 1790549143953521 us; error 48 us; skew 500 ppm
I20260927 22:45:43.953923 21870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:43 dist-test-slave-pp80 krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:43.965054 21870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:43.966845 21870 webserver.cc:460] Webserver started at http://127.0.0.1:40809/ using document root <none> and password file <none>
I20260927 22:45:43.967648 21870 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:43.967763 21870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:43.968087 21870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:43.971268 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "261b036525934293aa33eb8ee0018fa9"
format_stamp: "Formatted at 2026-09-27 22:45:43 on dist-test-slave-pp80"
server_key: "c953a23b8352b8b00899ec2855a754e8"
server_key_iv: "f256592d244e4ca121134fdca5e66105"
server_key_version: "encryptionkey@0"
I20260927 22:45:43.972086 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "261b036525934293aa33eb8ee0018fa9"
format_stamp: "Formatted at 2026-09-27 22:45:43 on dist-test-slave-pp80"
server_key: "c953a23b8352b8b00899ec2855a754e8"
server_key_iv: "f256592d244e4ca121134fdca5e66105"
server_key_version: "encryptionkey@0"
I20260927 22:45:43.977626 21870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:45:43.981632 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:43.983063 21870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:43.983307 21870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "261b036525934293aa33eb8ee0018fa9"
format_stamp: "Formatted at 2026-09-27 22:45:43 on dist-test-slave-pp80"
server_key: "c953a23b8352b8b00899ec2855a754e8"
server_key_iv: "f256592d244e4ca121134fdca5e66105"
server_key_version: "encryptionkey@0"
I20260927 22:45:43.983501 21870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:43.989640 21870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:43.994326 21870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:43.994625 21870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:44.014482 21870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35411
I20260927 22:45:44.014536 21936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35411 every 8 connection(s)
I20260927 22:45:44.015978 21870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:44.017184 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21870
I20260927 22:45:44.017369 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:44.017871 1237 external_mini_cluster.cc:1428] Setting key e3798811a978929a22b3c6027f8d7ec2
I20260927 22:45:44.020897 21937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:44.026182 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:44.019684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53645 (local address 127.0.0.1:35411)
0927 22:45:44.020537 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:44.020549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:44.020592 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:45:44.022393 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:44.022416 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:44.022761 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:44.023163 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:44.024285 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:44.024295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:44.024312 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:44.024337 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:44.024411 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:44.024441 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:44.024689 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:44.024831 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:45:44.025016 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":336,"threads_started":1}
I20260927 22:45:44.030488 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:44.033331 21937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9: Bootstrap starting.
I20260927 22:45:44.037343 21937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:44.037528 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:44.034990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54193 (local address 127.0.0.1:35411)
0927 22:45:44.035181 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:44.035186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:44.035504 (+ 318us) server_negotiation.cc:407] Connection header received
0927 22:45:44.035700 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:44.035704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:44.035752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:44.035824 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:44.036795 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:44.036799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:44.036802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:44.036817 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:44.036854 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:44.036868 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:44.036953 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:44.037101 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:45:44.037148 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 22:45:44.038614 21937 log.cc:824] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:44.041548 21937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9: No bootstrap required, opened a new log
I20260927 22:45:44.048925 21937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261b036525934293aa33eb8ee0018fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 22:45:44.049384 21937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:44.049512 21937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261b036525934293aa33eb8ee0018fa9, State: Initialized, Role: FOLLOWER
I20260927 22:45:44.050088 21937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261b036525934293aa33eb8ee0018fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 22:45:44.050331 21937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:44.050484 21937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:44.050663 21937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:44.053012 21937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261b036525934293aa33eb8ee0018fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 22:45:44.053557 21937 leader_election.cc:302] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261b036525934293aa33eb8ee0018fa9; no voters:
I20260927 22:45:44.054193 21937 leader_election.cc:288] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:44.054389 21947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:44.055826 21947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 261b036525934293aa33eb8ee0018fa9, State: Running, Role: LEADER
I20260927 22:45:44.056623 21937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:44.056572 21947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261b036525934293aa33eb8ee0018fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 22:45:44.060099 21948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261b036525934293aa33eb8ee0018fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261b036525934293aa33eb8ee0018fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } } }
I20260927 22:45:44.060030 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261b036525934293aa33eb8ee0018fa9. Latest consensus state: current_term: 1 leader_uuid: "261b036525934293aa33eb8ee0018fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261b036525934293aa33eb8ee0018fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } } }
I20260927 22:45:44.060621 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:44.060618 21948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:44.062387 21956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:44.067375 21956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:44.077284 21956 catalog_manager.cc:1347] Generated new cluster ID: 5d5f2c8724df454f86632261e282a8c3
I20260927 22:45:44.077430 21956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:44.093313 21956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:44.093801 21956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:44.107331 21956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261b036525934293aa33eb8ee0018fa9: Generated new TSK 0
I20260927 22:45:44.108304 21956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:44.122759 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21870
W20260927 22:45:44.123875 21932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53645
W20260927 22:45:44.124698 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21870: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:45:44.124861 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21870
2026-09-27T22:45:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:44 dist-test-slave-pp80 krb5kdc[21968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:44 dist-test-slave-pp80 krb5kdc[21968](info): set up 2 sockets
Sep 27 22:45:44 dist-test-slave-pp80 krb5kdc[21968](info): commencing operation
krb5kdc: starting...
W20260927 22:45:45.276741 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:45 dist-test-slave-pp80 krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:45.544430 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:45.893296 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:45.893617 21981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:45.893800 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:45.901561 21981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:45.901691 21981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:45.901806 21981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:45.901932 21981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:45.901996 21981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:45.912177 21981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:45.913609 21981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:45.914934 21981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:45.928162 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:45.928117 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:45.928133 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:45.928898 21981 server_base.cc:1034] running on GCE node
I20260927 22:45:45.929791 21981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:45.931394 21981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:45.932624 21981 hybrid_clock.cc:648] HybridClock initialized: now 1790549145932559 us; error 62 us; skew 500 ppm
I20260927 22:45:45.932991 21981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:45 dist-test-slave-pp80 krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:45.936597 21981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:45.938146 21981 webserver.cc:460] Webserver started at http://127.0.0.1:38921/ using document root <none> and password file <none>
I20260927 22:45:45.938874 21981 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:45.939028 21981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:45.939342 21981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:45.942487 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "c938e31006284767ae837bfe5bc01f19"
format_stamp: "Formatted at 2026-09-27 22:45:45 on dist-test-slave-pp80"
server_key: "9359e1fc175b6af41b132631981e95f1"
server_key_iv: "d1d4694d4d79f5330cd71d546e76e7a7"
server_key_version: "encryptionkey@0"
I20260927 22:45:45.943282 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "c938e31006284767ae837bfe5bc01f19"
format_stamp: "Formatted at 2026-09-27 22:45:45 on dist-test-slave-pp80"
server_key: "9359e1fc175b6af41b132631981e95f1"
server_key_iv: "d1d4694d4d79f5330cd71d546e76e7a7"
server_key_version: "encryptionkey@0"
I20260927 22:45:45.948949 21981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:45:45.952770 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:45.954231 21981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:45:45.954427 21981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "c938e31006284767ae837bfe5bc01f19"
format_stamp: "Formatted at 2026-09-27 22:45:45 on dist-test-slave-pp80"
server_key: "9359e1fc175b6af41b132631981e95f1"
server_key_iv: "d1d4694d4d79f5330cd71d546e76e7a7"
server_key_version: "encryptionkey@0"
I20260927 22:45:45.954636 21981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:45.971088 21981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:45.975874 21981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:45.976306 21981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:45.996029 21981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43139
I20260927 22:45:45.996127 22047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43139 every 8 connection(s)
I20260927 22:45:45.997555 21981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:46.001721 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 21981
I20260927 22:45:46.001757 22048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:46.001951 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:46.002458 1237 external_mini_cluster.cc:1428] Setting key b973cbd63d7140de31390c1bb234bfdb
I20260927 22:45:46.017235 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19: Bootstrap starting.
Sep 27 22:45:46 dist-test-slave-pp80 krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:45:46.021263 22048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:46.022528 22048 log.cc:824] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:46.025786 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19: No bootstrap required, opened a new log
I20260927 22:45:46.028375 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:46.004535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49909 (local address 127.0.0.1:43139)
0927 22:45:46.005491 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:46.005504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:46.005549 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:45:46.007930 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:46.007959 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:46.008494 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:46.008861 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:46.010900 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:46.012209 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:46.014347 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:46.015045 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:46.015196 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:46.020534 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:46.020543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:46.020560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:46.020607 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:46.023970 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:46.024803 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:46.024811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:46.024817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:46.024992 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:46.025418 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:46.025422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:46.025424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:46.026003 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:46.026352 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:46.026803 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 22:45:46.027252 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":479,"threads_started":1}
I20260927 22:45:46.032398 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:46.034886 22048 raft_consensus.cc:348] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938e31006284767ae837bfe5bc01f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260927 22:45:46.035437 22048 raft_consensus.cc:374] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:46.035591 22048 raft_consensus.cc:729] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c938e31006284767ae837bfe5bc01f19, State: Initialized, Role: FOLLOWER
I20260927 22:45:46.036329 22048 consensus_queue.cc:260] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938e31006284767ae837bfe5bc01f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260927 22:45:46.036623 22048 raft_consensus.cc:388] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:46.036793 22048 raft_consensus.cc:482] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:46.036983 22048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:46.040016 22048 raft_consensus.cc:504] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938e31006284767ae837bfe5bc01f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260927 22:45:46.040673 22048 leader_election.cc:302] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c938e31006284767ae837bfe5bc01f19; no voters:
I20260927 22:45:46.041515 22048 leader_election.cc:288] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:46.041685 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:46.044727 22048 sys_catalog.cc:564] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:46.048811 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [term 1 LEADER]: Becoming Leader. State: Replica: c938e31006284767ae837bfe5bc01f19, State: Running, Role: LEADER
I20260927 22:45:46.049715 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938e31006284767ae837bfe5bc01f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260927 22:45:46.053288 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c938e31006284767ae837bfe5bc01f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938e31006284767ae837bfe5bc01f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } } }
I20260927 22:45:46.053745 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:46.055456 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:46.037783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39797 (local address 127.0.0.1:43139)
0927 22:45:46.038048 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:46.038056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:46.038544 (+ 488us) server_negotiation.cc:407] Connection header received
0927 22:45:46.038792 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:46.038798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:46.038875 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:46.039002 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:46.040638 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:46.041468 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:46.043215 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:46.043797 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:46.043945 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:46.046694 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:46.046704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:46.046711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:46.046753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:46.051394 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:46.052366 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:46.052373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:46.052377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:46.052584 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:46.053127 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:46.053135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:46.053141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:46.053683 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:46.053988 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:46.054748 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 22:45:46.055036 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:45:46.055545 22072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:46.055361 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c938e31006284767ae837bfe5bc01f19. Latest consensus state: current_term: 1 leader_uuid: "c938e31006284767ae837bfe5bc01f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938e31006284767ae837bfe5bc01f19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } } }
I20260927 22:45:46.055752 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:46.060626 22072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:46.069525 22072 catalog_manager.cc:1347] Generated new cluster ID: 00e1ea7661fd42af8a87b469d304c618
I20260927 22:45:46.069667 22072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:46.096869 22072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:46.098472 22072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:46.109421 22072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c938e31006284767ae837bfe5bc01f19: Generated new TSK 0
I20260927 22:45:46.110332 22072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:46.117647 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 21981
I20260927 22:45:46.118830 22043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:46.212371 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 21981
2026-09-27T22:45:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:46 dist-test-slave-pp80 krb5kdc[22080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:46 dist-test-slave-pp80 krb5kdc[22080](info): set up 2 sockets
Sep 27 22:45:46 dist-test-slave-pp80 krb5kdc[22080](info): commencing operation
krb5kdc: starting...
W20260927 22:45:47.328708 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:47 dist-test-slave-pp80 krb5kdc[22080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:47.604971 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:47.956847 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:47.957227 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:47.957394 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:47.964844 22093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:47.965026 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:47.965150 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:47.965268 22093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:47.965348 22093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:47.975694 22093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:47.977255 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:47.978644 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:47.991264 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:47.991207 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:47.991276 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:47.992316 22093 server_base.cc:1034] running on GCE node
I20260927 22:45:47.993183 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:47.994658 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:47.995911 22093 hybrid_clock.cc:648] HybridClock initialized: now 1790549147995843 us; error 56 us; skew 500 ppm
I20260927 22:45:47.996457 22093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:47 dist-test-slave-pp80 krb5kdc[22080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:47.999940 22093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:48.001662 22093 webserver.cc:460] Webserver started at http://127.0.0.1:34475/ using document root <none> and password file <none>
I20260927 22:45:48.002389 22093 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:48.002491 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:48.002799 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:48.006207 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "feee0e120bc74d52872c352f5fdad39c"
format_stamp: "Formatted at 2026-09-27 22:45:47 on dist-test-slave-pp80"
server_key: "deaf563078d1689b98226fcc934e9095"
server_key_iv: "0b5725e471f3ff79f7c245e9eebdeb07"
server_key_version: "encryptionkey@0"
I20260927 22:45:48.007063 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "feee0e120bc74d52872c352f5fdad39c"
format_stamp: "Formatted at 2026-09-27 22:45:47 on dist-test-slave-pp80"
server_key: "deaf563078d1689b98226fcc934e9095"
server_key_iv: "0b5725e471f3ff79f7c245e9eebdeb07"
server_key_version: "encryptionkey@0"
I20260927 22:45:48.012951 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:45:48.017019 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:48.018575 22093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:45:48.018791 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "feee0e120bc74d52872c352f5fdad39c"
format_stamp: "Formatted at 2026-09-27 22:45:47 on dist-test-slave-pp80"
server_key: "deaf563078d1689b98226fcc934e9095"
server_key_iv: "0b5725e471f3ff79f7c245e9eebdeb07"
server_key_version: "encryptionkey@0"
I20260927 22:45:48.019037 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:48.039186 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:48.044340 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:48.044661 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:48.064634 22093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35393
I20260927 22:45:48.064659 22159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35393 every 8 connection(s)
I20260927 22:45:48.066152 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:48.070061 22160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:48.071946 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22093
I20260927 22:45:48.072294 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:48.072745 1237 external_mini_cluster.cc:1428] Setting key f4857c1a52fb42b1b20845e6b964babf
I20260927 22:45:48.083154 22160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c: Bootstrap starting.
I20260927 22:45:48.087284 22160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:48.087869 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:48.074845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32847 (local address 127.0.0.1:35393)
0927 22:45:48.075783 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:48.075797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:48.075845 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:45:48.077711 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:48.077737 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:48.078187 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:48.078560 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:48.079243 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:48.080756 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:48.082759 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:48.083619 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:48.083776 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:48.085236 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:48.085244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:48.085260 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:48.085848 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:48.085935 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:48.085983 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:48.086084 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:48.086334 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:45:48.086650 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":404,"threads_started":1}
I20260927 22:45:48.088631 22160 log.cc:824] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:48.092188 22160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c: No bootstrap required, opened a new log
I20260927 22:45:48.092376 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:48.102448 22160 raft_consensus.cc:348] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feee0e120bc74d52872c352f5fdad39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35393 } }
I20260927 22:45:48.103047 22160 raft_consensus.cc:374] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:48.103199 22160 raft_consensus.cc:729] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feee0e120bc74d52872c352f5fdad39c, State: Initialized, Role: FOLLOWER
I20260927 22:45:48.104033 22160 consensus_queue.cc:260] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feee0e120bc74d52872c352f5fdad39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35393 } }
I20260927 22:45:48.104358 22160 raft_consensus.cc:388] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:48.104400 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:48.096934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43237 (local address 127.0.0.1:35393)
0927 22:45:48.097224 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:48.097233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:48.097633 (+ 400us) server_negotiation.cc:407] Connection header received
0927 22:45:48.097838 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:48.097845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:48.097916 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:48.098011 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:48.098605 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:48.099398 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:48.101436 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:48.101930 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:48.102123 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:48.103576 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:48.103582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:48.103587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:48.103614 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:48.103669 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:48.103691 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:48.103786 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:48.103933 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:45:48.104045 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:45:48.104540 22160 raft_consensus.cc:482] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:48.104743 22160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:48.107264 22160 raft_consensus.cc:504] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feee0e120bc74d52872c352f5fdad39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35393 } }
I20260927 22:45:48.107784 22160 leader_election.cc:302] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feee0e120bc74d52872c352f5fdad39c; no voters:
I20260927 22:45:48.108444 22160 leader_election.cc:288] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:48.108603 22170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:48.109354 22170 raft_consensus.cc:686] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [term 1 LEADER]: Becoming Leader. State: Replica: feee0e120bc74d52872c352f5fdad39c, State: Running, Role: LEADER
I20260927 22:45:48.109978 22170 consensus_queue.cc:237] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feee0e120bc74d52872c352f5fdad39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35393 } }
I20260927 22:45:48.110821 22160 sys_catalog.cc:564] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:48.113336 22172 sys_catalog.cc:455] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [sys.catalog]: SysCatalogTable state changed. Reason: New leader feee0e120bc74d52872c352f5fdad39c. Latest consensus state: current_term: 1 leader_uuid: "feee0e120bc74d52872c352f5fdad39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feee0e120bc74d52872c352f5fdad39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35393 } } }
I20260927 22:45:48.113706 22172 sys_catalog.cc:458] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:48.115407 22178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:48.115365 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feee0e120bc74d52872c352f5fdad39c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feee0e120bc74d52872c352f5fdad39c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35393 } } }
I20260927 22:45:48.115684 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:48.121527 22178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:48.132151 22178 catalog_manager.cc:1347] Generated new cluster ID: 2ce9166819234ce3ae9eb4c9b8e8d125
I20260927 22:45:48.132328 22178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:48.144129 22178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:48.146147 22178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:48.159230 22178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feee0e120bc74d52872c352f5fdad39c: Generated new TSK 0
I20260927 22:45:48.160326 22178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:48.167572 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22093
W20260927 22:45:48.168969 22155 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32847
W20260927 22:45:48.169852 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22093: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:45:48.170032 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22093
2026-09-27T22:45:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:48 dist-test-slave-pp80 krb5kdc[22191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:48 dist-test-slave-pp80 krb5kdc[22191](info): set up 2 sockets
Sep 27 22:45:48 dist-test-slave-pp80 krb5kdc[22191](info): commencing operation
krb5kdc: starting...
W20260927 22:45:49.285063 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:49 dist-test-slave-pp80 krb5kdc[22191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:49.561738 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:49.910578 22204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:49.910887 22204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:49.911065 22204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:49.918515 22204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:49.918669 22204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:49.918761 22204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:49.918871 22204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:49.918953 22204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:49.929095 22204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:49.930475 22204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:49.931864 22204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:49.943909 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:49.944020 22212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:49.944020 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:49.944705 22204 server_base.cc:1034] running on GCE node
I20260927 22:45:49.945503 22204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:49.947032 22204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:49.948324 22204 hybrid_clock.cc:648] HybridClock initialized: now 1790549149948269 us; error 40 us; skew 500 ppm
I20260927 22:45:49.948711 22204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:49 dist-test-slave-pp80 krb5kdc[22191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:49.956591 22204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:49.958118 22204 webserver.cc:460] Webserver started at http://127.0.0.1:37843/ using document root <none> and password file <none>
I20260927 22:45:49.958860 22204 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:49.958966 22204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:49.959219 22204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:49.962294 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c"
format_stamp: "Formatted at 2026-09-27 22:45:49 on dist-test-slave-pp80"
server_key: "b9b3d2ea9bacee03b5e3de8cceb59a9e"
server_key_iv: "f426e48e770549b0070e4b1d28acc09f"
server_key_version: "encryptionkey@0"
I20260927 22:45:49.963016 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c"
format_stamp: "Formatted at 2026-09-27 22:45:49 on dist-test-slave-pp80"
server_key: "b9b3d2ea9bacee03b5e3de8cceb59a9e"
server_key_iv: "f426e48e770549b0070e4b1d28acc09f"
server_key_version: "encryptionkey@0"
I20260927 22:45:49.968430 22204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:45:49.972003 22219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:49.973604 22204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:49.973800 22204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c"
format_stamp: "Formatted at 2026-09-27 22:45:49 on dist-test-slave-pp80"
server_key: "b9b3d2ea9bacee03b5e3de8cceb59a9e"
server_key_iv: "f426e48e770549b0070e4b1d28acc09f"
server_key_version: "encryptionkey@0"
I20260927 22:45:49.974063 22204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:49.980211 22204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:49.984484 22204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:49.984800 22204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:50.004302 22204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260927 22:45:50.004357 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260927 22:45:50.005818 22204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:50.008882 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22204
I20260927 22:45:50.009096 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:50.009565 1237 external_mini_cluster.cc:1428] Setting key 9399f8c0b186c4299fc9f4a6e49fb0b4
I20260927 22:45:50.010002 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:50.019399 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:50.012014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52785 (local address 127.0.0.1:42683)
0927 22:45:50.012928 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:50.012950 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:50.013009 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:45:50.015175 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:50.015209 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:50.015727 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:50.016218 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:50.017324 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:50.017332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:50.017351 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:50.017368 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:50.017449 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:50.017480 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:50.017722 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:50.017868 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:45:50.018074 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":390,"threads_started":1}
I20260927 22:45:50.023223 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:50.024922 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c: Bootstrap starting.
I20260927 22:45:50.028584 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:50.029645 22271 log.cc:824] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:50.030517 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:50.027775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53707 (local address 127.0.0.1:42683)
0927 22:45:50.027949 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:50.027955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:50.028395 (+ 440us) server_negotiation.cc:407] Connection header received
0927 22:45:50.028591 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:50.028595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:50.028667 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:50.028798 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:50.029785 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:50.029791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:50.029795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:50.029813 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:50.029862 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:50.029883 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:50.030003 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:50.030134 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:45:50.030196 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:45:50.032779 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c: No bootstrap required, opened a new log
I20260927 22:45:50.040570 22271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260927 22:45:50.040997 22271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:50.041095 22271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5bfd9e29f547e585ef5c3bed6fc40c, State: Initialized, Role: FOLLOWER
I20260927 22:45:50.041775 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260927 22:45:50.042147 22271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:50.042308 22271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:50.042505 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:50.044862 22271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260927 22:45:50.045392 22271 leader_election.cc:302] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e5bfd9e29f547e585ef5c3bed6fc40c; no voters:
I20260927 22:45:50.046018 22271 leader_election.cc:288] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:50.046211 22281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:50.046893 22281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [term 1 LEADER]: Becoming Leader. State: Replica: 8e5bfd9e29f547e585ef5c3bed6fc40c, State: Running, Role: LEADER
I20260927 22:45:50.047495 22281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260927 22:45:50.048306 22271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:50.050691 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e5bfd9e29f547e585ef5c3bed6fc40c. Latest consensus state: current_term: 1 leader_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260927 22:45:50.051481 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:50.050994 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e5bfd9e29f547e585ef5c3bed6fc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260927 22:45:50.051887 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:50.055006 22290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:50.060863 22290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:50.070307 22290 catalog_manager.cc:1347] Generated new cluster ID: 462706e1215a4363919b21b4a6f0e8b4
I20260927 22:45:50.070451 22290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:50.092826 22290 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:50.093344 22290 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:50.103340 22290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e5bfd9e29f547e585ef5c3bed6fc40c: Generated new TSK 0
I20260927 22:45:50.104357 22290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:50.116163 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22204
W20260927 22:45:50.117522 22266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52785
W20260927 22:45:50.118558 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:45:50.118768 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22204
2026-09-27T22:45:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:50 dist-test-slave-pp80 krb5kdc[22302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:50 dist-test-slave-pp80 krb5kdc[22302](info): set up 2 sockets
Sep 27 22:45:50 dist-test-slave-pp80 krb5kdc[22302](info): commencing operation
krb5kdc: starting...
W20260927 22:45:51.236737 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:51 dist-test-slave-pp80 krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:51.509399 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:51.873548 22315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:51.873880 22315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:51.874045 22315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:51.881515 22315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:51.881649 22315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:51.881749 22315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:51.881855 22315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:51.881935 22315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:51.892215 22315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:51.893633 22315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:51.895004 22315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:51.907172 22320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:51.907284 22321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:51.907251 22323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:51.907986 22315 server_base.cc:1034] running on GCE node
I20260927 22:45:51.908927 22315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:51.910544 22315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:51.911806 22315 hybrid_clock.cc:648] HybridClock initialized: now 1790549151911777 us; error 63 us; skew 500 ppm
I20260927 22:45:51.912415 22315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:51 dist-test-slave-pp80 krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:51.915757 22315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:51.917625 22315 webserver.cc:460] Webserver started at http://127.0.0.1:39431/ using document root <none> and password file <none>
I20260927 22:45:51.918430 22315 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:51.918591 22315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:51.918932 22315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:51.922369 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "1c5e854b66c04c92ae439f9df628fd54"
format_stamp: "Formatted at 2026-09-27 22:45:51 on dist-test-slave-pp80"
server_key: "e576e559e570316294cc6851db7fca91"
server_key_iv: "bb5f6c58ce01418f6ea92ad07df03439"
server_key_version: "encryptionkey@0"
I20260927 22:45:51.923270 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "1c5e854b66c04c92ae439f9df628fd54"
format_stamp: "Formatted at 2026-09-27 22:45:51 on dist-test-slave-pp80"
server_key: "e576e559e570316294cc6851db7fca91"
server_key_iv: "bb5f6c58ce01418f6ea92ad07df03439"
server_key_version: "encryptionkey@0"
I20260927 22:45:51.928892 22315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:45:51.932484 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:51.933827 22315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:45:51.934023 22315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "1c5e854b66c04c92ae439f9df628fd54"
format_stamp: "Formatted at 2026-09-27 22:45:51 on dist-test-slave-pp80"
server_key: "e576e559e570316294cc6851db7fca91"
server_key_iv: "bb5f6c58ce01418f6ea92ad07df03439"
server_key_version: "encryptionkey@0"
I20260927 22:45:51.934237 22315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:51.946551 22315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:51.951045 22315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:51.951345 22315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:51.971001 22315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35157
I20260927 22:45:51.971027 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35157 every 8 connection(s)
I20260927 22:45:51.972514 22315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:51.976105 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22315
I20260927 22:45:51.976357 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:51.976663 22382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:51.976858 1237 external_mini_cluster.cc:1428] Setting key cf5ccf73cf5a1b48bee6427bf155e0bb
I20260927 22:45:51.992199 22382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54: Bootstrap starting.
Sep 27 22:45:51 dist-test-slave-pp80 krb5kdc[22302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:45:51.995774 22382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:51.996984 22382 log.cc:824] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:52.000653 22382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54: No bootstrap required, opened a new log
I20260927 22:45:52.002775 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:51.978910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:58327 (local address 127.0.0.1:35157)
0927 22:45:51.979803 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:51.979814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:51.979855 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:45:51.981677 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:51.981703 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:51.982163 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:51.982538 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:51.984415 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:51.985756 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:51.987670 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:51.988582 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:51.988727 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:51.993868 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:51.993876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:51.993891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:51.993947 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:51.997887 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:51.999063 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:51.999070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:51.999076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:51.999248 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:51.999781 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:51.999785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:51.999787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:52.000473 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:52.000706 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:52.001342 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 22:45:52.001716 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":386,"threads_started":1}
I20260927 22:45:52.006908 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:52.010306 22382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e854b66c04c92ae439f9df628fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260927 22:45:52.010895 22382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:52.011022 22382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5e854b66c04c92ae439f9df628fd54, State: Initialized, Role: FOLLOWER
I20260927 22:45:52.011811 22382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e854b66c04c92ae439f9df628fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260927 22:45:52.012109 22382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:52.012351 22382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:52.012614 22382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:52.016094 22382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e854b66c04c92ae439f9df628fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260927 22:45:52.016821 22382 leader_election.cc:302] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c5e854b66c04c92ae439f9df628fd54; no voters:
I20260927 22:45:52.017570 22382 leader_election.cc:288] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:52.017694 22392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:52.020560 22392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [term 1 LEADER]: Becoming Leader. State: Replica: 1c5e854b66c04c92ae439f9df628fd54, State: Running, Role: LEADER
I20260927 22:45:52.020571 22382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:52.021147 22392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e854b66c04c92ae439f9df628fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260927 22:45:52.025229 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c5e854b66c04c92ae439f9df628fd54. Latest consensus state: current_term: 1 leader_uuid: "1c5e854b66c04c92ae439f9df628fd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e854b66c04c92ae439f9df628fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } } }
I20260927 22:45:52.025700 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:52.026403 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c5e854b66c04c92ae439f9df628fd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5e854b66c04c92ae439f9df628fd54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } } }
I20260927 22:45:52.026803 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:52.032228 22401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:52.033344 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:52.012502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:56807 (local address 127.0.0.1:35157)
0927 22:45:52.012752 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:52.012761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:52.013250 (+ 489us) server_negotiation.cc:407] Connection header received
0927 22:45:52.013438 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:52.013442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:52.013512 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:52.013614 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:52.015557 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:52.016413 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:52.018300 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:52.018990 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:52.019103 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:52.022888 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:52.022896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:52.022902 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:52.022956 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:52.027474 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:52.028982 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:52.028991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:52.028997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:52.029341 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:52.030093 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:52.030098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:52.030102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:52.030600 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:52.032582 (+ 1982us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:52.032761 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 22:45:52.032939 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:45:52.037932 22401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:52.048561 22401 catalog_manager.cc:1347] Generated new cluster ID: 0607d41c00f44a659cba8041676e5ac5
I20260927 22:45:52.048727 22401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:52.065624 22401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:52.067618 22401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:52.081126 22401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c5e854b66c04c92ae439f9df628fd54: Generated new TSK 0
I20260927 22:45:52.082296 22401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:52.090806 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22315
I20260927 22:45:52.092192 22377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:52.186417 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22315
W20260927 22:45:52.205842 22300 connection.cc:535] client connection to 127.0.0.1:35157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35157): Connection reset by peer (error 104)
2026-09-27T22:45:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:52 dist-test-slave-pp80 krb5kdc[22415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:52 dist-test-slave-pp80 krb5kdc[22415](info): set up 2 sockets
Sep 27 22:45:52 dist-test-slave-pp80 krb5kdc[22415](info): commencing operation
krb5kdc: starting...
W20260927 22:45:53.304939 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:53 dist-test-slave-pp80 krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:53.576192 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:53.938501 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:53.938874 22428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:53.939047 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:53.946401 22428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:53.946573 22428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:53.946682 22428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:53.946794 22428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:53.946877 22428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:53.956944 22428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:53.958400 22428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:53.959776 22428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:53.972121 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:53.972190 22436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:53.972165 22434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:53.972957 22428 server_base.cc:1034] running on GCE node
I20260927 22:45:53.973793 22428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:53.975263 22428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:53.976461 22428 hybrid_clock.cc:648] HybridClock initialized: now 1790549153976400 us; error 62 us; skew 500 ppm
I20260927 22:45:53.976892 22428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:53 dist-test-slave-pp80 krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:53.980037 22428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:53.981582 22428 webserver.cc:460] Webserver started at http://127.0.0.1:33011/ using document root <none> and password file <none>
I20260927 22:45:53.982352 22428 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:53.982502 22428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:53.982822 22428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:53.985912 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "58e1428461724cf49c4b9348b2fb7fc9"
format_stamp: "Formatted at 2026-09-27 22:45:53 on dist-test-slave-pp80"
server_key: "7ce5b755b16e8b48b4821cba843692dd"
server_key_iv: "01461aec1dc8809c4fe5e2976e74e73b"
server_key_version: "encryptionkey@0"
I20260927 22:45:53.986757 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "58e1428461724cf49c4b9348b2fb7fc9"
format_stamp: "Formatted at 2026-09-27 22:45:53 on dist-test-slave-pp80"
server_key: "7ce5b755b16e8b48b4821cba843692dd"
server_key_iv: "01461aec1dc8809c4fe5e2976e74e73b"
server_key_version: "encryptionkey@0"
I20260927 22:45:53.992077 22428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:45:53.995726 22443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:53.997274 22428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:53.997462 22428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "58e1428461724cf49c4b9348b2fb7fc9"
format_stamp: "Formatted at 2026-09-27 22:45:53 on dist-test-slave-pp80"
server_key: "7ce5b755b16e8b48b4821cba843692dd"
server_key_iv: "01461aec1dc8809c4fe5e2976e74e73b"
server_key_version: "encryptionkey@0"
I20260927 22:45:53.997673 22428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:54.017355 22428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:54.021881 22428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:54.022186 22428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:54.041628 22428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43787
I20260927 22:45:54.041689 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43787 every 8 connection(s)
I20260927 22:45:54.043099 22428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:54.043637 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22428
I20260927 22:45:54.043836 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:54.044493 1237 external_mini_cluster.cc:1428] Setting key 56cf9d7f9b44a1629ea83690ae1cb8f7
I20260927 22:45:54.047734 22495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:54.058290 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:54.046458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:49345 (local address 127.0.0.1:43787)
0927 22:45:54.047250 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:54.047263 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:54.047303 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:45:54.049109 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:54.049132 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:54.049537 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:54.049933 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:54.050519 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:54.051912 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:54.053428 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:54.054123 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:54.054339 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:54.055737 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:54.055746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:54.055765 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:54.056467 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:54.056559 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:54.056588 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:54.056784 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:54.057057 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:45:54.057250 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":352,"threads_started":1}
I20260927 22:45:54.059397 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9: Bootstrap starting.
I20260927 22:45:54.062147 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:54.062870 22495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:54.063905 22495 log.cc:824] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:54.067341 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9: No bootstrap required, opened a new log
I20260927 22:45:54.073522 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:54.066610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:57325 (local address 127.0.0.1:43787)
0927 22:45:54.066810 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:54.066816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:54.067341 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:45:54.067501 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:54.067506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:54.067582 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:54.067692 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:54.068141 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:54.069269 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:54.070604 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:54.071352 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:54.071488 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:54.072662 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:54.072669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:54.072674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:54.072705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:54.072779 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:54.072803 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:54.072928 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:54.073089 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 22:45:54.073160 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:45:54.076992 22495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e1428461724cf49c4b9348b2fb7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43787 } }
I20260927 22:45:54.077409 22495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:54.077561 22495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e1428461724cf49c4b9348b2fb7fc9, State: Initialized, Role: FOLLOWER
I20260927 22:45:54.078186 22495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e1428461724cf49c4b9348b2fb7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43787 } }
I20260927 22:45:54.078418 22495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:54.078573 22495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:54.078775 22495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:54.081126 22495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e1428461724cf49c4b9348b2fb7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43787 } }
I20260927 22:45:54.081759 22495 leader_election.cc:302] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e1428461724cf49c4b9348b2fb7fc9; no voters:
I20260927 22:45:54.082386 22495 leader_election.cc:288] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:54.082531 22505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:54.083235 22505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 58e1428461724cf49c4b9348b2fb7fc9, State: Running, Role: LEADER
I20260927 22:45:54.083782 22505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e1428461724cf49c4b9348b2fb7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43787 } }
I20260927 22:45:54.084573 22495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:54.086750 22506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e1428461724cf49c4b9348b2fb7fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e1428461724cf49c4b9348b2fb7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43787 } } }
I20260927 22:45:54.087198 22506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:54.088749 22514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:54.088590 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e1428461724cf49c4b9348b2fb7fc9. Latest consensus state: current_term: 1 leader_uuid: "58e1428461724cf49c4b9348b2fb7fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e1428461724cf49c4b9348b2fb7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43787 } } }
I20260927 22:45:54.088961 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:54.096220 22514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:54.105211 22514 catalog_manager.cc:1347] Generated new cluster ID: 7466f58f15f54175b77c887ec00a36fe
I20260927 22:45:54.105350 22514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:54.143568 22514 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:54.145030 22514 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:54.152073 22514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e1428461724cf49c4b9348b2fb7fc9: Generated new TSK 0
I20260927 22:45:54.153244 22514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:54.159953 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22428
W20260927 22:45:54.161521 22490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.206:49345
W20260927 22:45:54.162310 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22428: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:45:54.162472 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22428
W20260927 22:45:54.179926 22413 connection.cc:535] client connection to 127.0.0.1:43787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43787): Connection reset by peer (error 104)
2026-09-27T22:45:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:54 dist-test-slave-pp80 krb5kdc[22526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:54 dist-test-slave-pp80 krb5kdc[22526](info): set up 2 sockets
Sep 27 22:45:54 dist-test-slave-pp80 krb5kdc[22526](info): commencing operation
krb5kdc: starting...
W20260927 22:45:55.280776 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:55 dist-test-slave-pp80 krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:55.547744 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:55.901135 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:55.901536 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:55.901707 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:55.909528 22539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:55.909727 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:55.909850 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:55.909960 22539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:55.910050 22539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:55.920627 22539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:55.922147 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:55.923576 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:55.937300 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:55.937422 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:55.937559 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:55.938127 22539 server_base.cc:1034] running on GCE node
I20260927 22:45:55.939010 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:55.940725 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:55.941941 22539 hybrid_clock.cc:648] HybridClock initialized: now 1790549155941893 us; error 39 us; skew 500 ppm
I20260927 22:45:55.942369 22539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:55 dist-test-slave-pp80 krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:55.954255 22539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:55.956017 22539 webserver.cc:460] Webserver started at http://127.0.0.1:40141/ using document root <none> and password file <none>
I20260927 22:45:55.956874 22539 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:55.957039 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:55.957373 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:55.960632 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "60307eae751340d4958e5d72076a1210"
format_stamp: "Formatted at 2026-09-27 22:45:55 on dist-test-slave-pp80"
server_key: "3809dcdb8f8ad51a8c85762e96841e89"
server_key_iv: "a22402bb1c0a7b1f46ace3ae62e89ed7"
server_key_version: "encryptionkey@0"
I20260927 22:45:55.961464 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "60307eae751340d4958e5d72076a1210"
format_stamp: "Formatted at 2026-09-27 22:45:55 on dist-test-slave-pp80"
server_key: "3809dcdb8f8ad51a8c85762e96841e89"
server_key_iv: "a22402bb1c0a7b1f46ace3ae62e89ed7"
server_key_version: "encryptionkey@0"
I20260927 22:45:55.967182 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:45:55.971181 22554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:55.972747 22539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:45:55.972960 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "60307eae751340d4958e5d72076a1210"
format_stamp: "Formatted at 2026-09-27 22:45:55 on dist-test-slave-pp80"
server_key: "3809dcdb8f8ad51a8c85762e96841e89"
server_key_iv: "a22402bb1c0a7b1f46ace3ae62e89ed7"
server_key_version: "encryptionkey@0"
I20260927 22:45:55.973201 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:55.979691 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:55.984481 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:55.984798 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:56.004472 22539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33653
I20260927 22:45:56.004489 22605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33653 every 8 connection(s)
I20260927 22:45:56.006079 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:56.010134 22606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:45:56.015316 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22539
I20260927 22:45:56.015477 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:56.015993 1237 external_mini_cluster.cc:1428] Setting key 1223f6f1a5a0ff30a6af5c04bcae34a3
I20260927 22:45:56.024232 22606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210: Bootstrap starting.
I20260927 22:45:56.026260 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:56.018143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:43563 (local address 127.0.0.1:33653)
0927 22:45:56.019049 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:56.019067 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:56.019131 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:45:56.022085 (+ 2954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:56.022119 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:56.022640 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:56.023127 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:56.024161 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:56.024168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:56.024186 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:56.024205 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:56.024319 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:56.024350 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:56.024581 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:56.024714 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:45:56.024925 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":392,"threads_started":1}
I20260927 22:45:56.028079 22606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:56.029377 22606 log.cc:824] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:56.030436 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:56.032666 22606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210: No bootstrap required, opened a new log
I20260927 22:45:56.037629 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:56.034952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:51545 (local address 127.0.0.1:33653)
0927 22:45:56.035135 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:56.035143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:56.035559 (+ 416us) server_negotiation.cc:407] Connection header received
0927 22:45:56.035760 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:56.035776 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:56.035831 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:56.035940 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:56.036897 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:56.036903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:56.036908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:45:56.036925 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:56.036973 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:45:56.036990 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:56.037135 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:56.037256 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:45:56.037329 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:45:56.041602 22606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60307eae751340d4958e5d72076a1210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260927 22:45:56.042140 22606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:56.042263 22606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60307eae751340d4958e5d72076a1210, State: Initialized, Role: FOLLOWER
I20260927 22:45:56.042876 22606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60307eae751340d4958e5d72076a1210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260927 22:45:56.043151 22606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:56.043308 22606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:56.043495 22606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:56.046191 22606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60307eae751340d4958e5d72076a1210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260927 22:45:56.046746 22606 leader_election.cc:302] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60307eae751340d4958e5d72076a1210; no voters:
I20260927 22:45:56.047340 22606 leader_election.cc:288] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:56.047482 22616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:56.048281 22616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [term 1 LEADER]: Becoming Leader. State: Replica: 60307eae751340d4958e5d72076a1210, State: Running, Role: LEADER
I20260927 22:45:56.048812 22616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60307eae751340d4958e5d72076a1210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260927 22:45:56.049562 22606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:56.052137 22617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60307eae751340d4958e5d72076a1210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60307eae751340d4958e5d72076a1210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } } }
I20260927 22:45:56.052649 22617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:56.052507 22618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60307eae751340d4958e5d72076a1210. Latest consensus state: current_term: 1 leader_uuid: "60307eae751340d4958e5d72076a1210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60307eae751340d4958e5d72076a1210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } } }
I20260927 22:45:56.052805 22618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:56.055344 22622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:56.060883 22622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:56.071055 22622 catalog_manager.cc:1347] Generated new cluster ID: dee632bae013426ab00efd8175ff562e
I20260927 22:45:56.071192 22622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:56.094789 22622 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:56.095316 22622 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:56.105158 22622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60307eae751340d4958e5d72076a1210: Generated new TSK 0
I20260927 22:45:56.106467 22622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:56.111300 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22539
W20260927 22:45:56.112912 22601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.206:43563
W20260927 22:45:56.113703 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22539: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:45:56.113914 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22539
2026-09-27T22:45:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:56 dist-test-slave-pp80 krb5kdc[22637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:56 dist-test-slave-pp80 krb5kdc[22637](info): set up 2 sockets
Sep 27 22:45:56 dist-test-slave-pp80 krb5kdc[22637](info): commencing operation
krb5kdc: starting...
W20260927 22:45:57.268692 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:57 dist-test-slave-pp80 krb5kdc[22637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:57.540315 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:57.884497 22650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:57.884831 22650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:57.885001 22650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:57.892485 22650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:57.892608 22650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:57.892709 22650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:57.892814 22650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:57.892899 22650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:57.903306 22650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:57.904729 22650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:57.906145 22650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:57.918846 22655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:57.918974 22658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:57.919021 22656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:57.920194 22650 server_base.cc:1034] running on GCE node
I20260927 22:45:57.921176 22650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:57.922883 22650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:57.924161 22650 hybrid_clock.cc:648] HybridClock initialized: now 1790549157924123 us; error 62 us; skew 500 ppm
I20260927 22:45:57.924633 22650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:57 dist-test-slave-pp80 krb5kdc[22637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:57.928129 22650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:57.929684 22650 webserver.cc:460] Webserver started at http://127.0.0.1:34327/ using document root <none> and password file <none>
I20260927 22:45:57.930404 22650 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:57.930558 22650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:57.930871 22650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:57.933962 22650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "df8cfdf03f534c46b141b9fd35e072c6"
format_stamp: "Formatted at 2026-09-27 22:45:57 on dist-test-slave-pp80"
server_key: "2b3557979bee981dbd24c68cd0bf4f4f"
server_key_iv: "2082bc7276acf7560d7d1646cb2f993e"
server_key_version: "encryptionkey@0"
I20260927 22:45:57.934756 22650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "df8cfdf03f534c46b141b9fd35e072c6"
format_stamp: "Formatted at 2026-09-27 22:45:57 on dist-test-slave-pp80"
server_key: "2b3557979bee981dbd24c68cd0bf4f4f"
server_key_iv: "2082bc7276acf7560d7d1646cb2f993e"
server_key_version: "encryptionkey@0"
I20260927 22:45:57.940222 22650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:45:57.943920 22665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:45:57.945464 22650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:45:57.945674 22650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "df8cfdf03f534c46b141b9fd35e072c6"
format_stamp: "Formatted at 2026-09-27 22:45:57 on dist-test-slave-pp80"
server_key: "2b3557979bee981dbd24c68cd0bf4f4f"
server_key_iv: "2082bc7276acf7560d7d1646cb2f993e"
server_key_version: "encryptionkey@0"
I20260927 22:45:57.945935 22650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:45:57.958546 22650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:45:57.962847 22650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:45:57.963136 22650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:45:57.982772 22650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35673
I20260927 22:45:57.982817 22716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35673 every 8 connection(s)
I20260927 22:45:57.984257 22650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:45:57.986891 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22650
I20260927 22:45:57.987066 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:45:57.987602 1237 external_mini_cluster.cc:1428] Setting key 011f7dbdb1c4b237970eeca6fa956565
I20260927 22:45:57.988797 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:45:57 dist-test-slave-pp80 krb5kdc[22637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:45:58.003633 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6: Bootstrap starting.
I20260927 22:45:58.007328 22717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6: Neither blocks nor log segments found. Creating new log.
I20260927 22:45:58.008467 22717 log.cc:824] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:45:58.011583 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6: No bootstrap required, opened a new log
I20260927 22:45:58.011737 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:57.989465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:46773 (local address 127.0.0.1:35673)
0927 22:45:57.990221 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:57.990233 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:57.990273 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:45:57.991864 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:57.991887 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:57.992383 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:57.992764 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:57.994604 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:57.996061 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:57.997878 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:57.998585 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:57.998737 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:58.003732 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:58.003741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:58.003759 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:58.003833 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:58.007054 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:58.008000 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:58.008008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:58.008015 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:58.008200 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:58.008765 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:58.008771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:58.008774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:58.009494 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:58.009844 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:58.010367 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 22:45:58.010699 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":343,"threads_started":1}
I20260927 22:45:58.015935 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:45:58.020571 22717 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8cfdf03f534c46b141b9fd35e072c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35673 } }
I20260927 22:45:58.021111 22717 raft_consensus.cc:374] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:45:58.021240 22717 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8cfdf03f534c46b141b9fd35e072c6, State: Initialized, Role: FOLLOWER
I20260927 22:45:58.021922 22717 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8cfdf03f534c46b141b9fd35e072c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35673 } }
I20260927 22:45:58.022161 22717 raft_consensus.cc:388] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:45:58.022322 22717 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:45:58.022504 22717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:45:58.025007 22717 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8cfdf03f534c46b141b9fd35e072c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35673 } }
I20260927 22:45:58.025588 22717 leader_election.cc:302] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df8cfdf03f534c46b141b9fd35e072c6; no voters:
I20260927 22:45:58.026366 22717 leader_election.cc:288] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:45:58.026492 22727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:45:58.027318 22727 raft_consensus.cc:686] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [term 1 LEADER]: Becoming Leader. State: Replica: df8cfdf03f534c46b141b9fd35e072c6, State: Running, Role: LEADER
I20260927 22:45:58.028079 22727 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8cfdf03f534c46b141b9fd35e072c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35673 } }
I20260927 22:45:58.030270 22717 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:45:58.031293 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8cfdf03f534c46b141b9fd35e072c6. Latest consensus state: current_term: 1 leader_uuid: "df8cfdf03f534c46b141b9fd35e072c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8cfdf03f534c46b141b9fd35e072c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35673 } } }
I20260927 22:45:58.031692 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:58.039602 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:45:58.020411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:45017 (local address 127.0.0.1:35673)
0927 22:45:58.020663 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:45:58.020670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:45:58.021024 (+ 354us) server_negotiation.cc:407] Connection header received
0927 22:45:58.021247 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:45:58.021259 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:45:58.021347 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:45:58.021476 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:45:58.023179 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:58.024033 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:58.025382 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:45:58.025887 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:45:58.026034 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:45:58.029465 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:45:58.029470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:45:58.029473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:45:58.029515 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:45:58.033977 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:58.035407 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:58.035416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:58.035422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:58.035649 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:45:58.036401 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:45:58.036410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:45:58.036416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:45:58.036925 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:45:58.038536 (+ 1611us) server_negotiation.cc:1035] Waiting for connection context
0927 22:45:58.038767 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 22:45:58.039018 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:45:58.041376 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df8cfdf03f534c46b141b9fd35e072c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8cfdf03f534c46b141b9fd35e072c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35673 } } }
I20260927 22:45:58.042310 22732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:45:58.042310 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:45:58.048198 22732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:45:58.057950 22732 catalog_manager.cc:1347] Generated new cluster ID: 18ca0fa60a914f5dbcfb0be1b26b2322
I20260927 22:45:58.058105 22732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:45:58.080842 22732 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:45:58.082193 22732 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:45:58.095093 22732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df8cfdf03f534c46b141b9fd35e072c6: Generated new TSK 0
I20260927 22:45:58.096050 22732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:45:58.102399 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22650
I20260927 22:45:58.103837 22712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:45:58.200150 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22650
W20260927 22:45:58.218082 22635 connection.cc:535] client connection to 127.0.0.1:35673 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35673): Connection reset by peer (error 104)
2026-09-27T22:45:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:45:58 dist-test-slave-pp80 krb5kdc[22749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:45:58 dist-test-slave-pp80 krb5kdc[22749](info): set up 2 sockets
Sep 27 22:45:58 dist-test-slave-pp80 krb5kdc[22749](info): commencing operation
krb5kdc: starting...
W20260927 22:45:59.320958 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:45:59 dist-test-slave-pp80 krb5kdc[22749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:45:59.597290 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:45:59.947190 22762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:45:59.947563 22762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:45:59.947743 22762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:45:59.955281 22762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:45:59.955454 22762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:45:59.955574 22762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:45:59.955683 22762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:45:59.955768 22762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:45:59.966224 22762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:45:59.967641 22762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:45:59.969038 22762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:45:59.981593 22767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:59.981575 22768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:45:59.981518 22770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:45:59.982301 22762 server_base.cc:1034] running on GCE node
I20260927 22:45:59.983170 22762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:45:59.984781 22762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:45:59.986053 22762 hybrid_clock.cc:648] HybridClock initialized: now 1790549159985987 us; error 60 us; skew 500 ppm
I20260927 22:45:59.986428 22762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:45:59 dist-test-slave-pp80 krb5kdc[22749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:45:59.989857 22762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:45:59.991431 22762 webserver.cc:460] Webserver started at http://127.0.0.1:42351/ using document root <none> and password file <none>
I20260927 22:45:59.992170 22762 fs_manager.cc:359] Metadata directory not provided
I20260927 22:45:59.992357 22762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:45:59.992681 22762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:45:59.995749 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "cd3039483a1b4d22b05c6b14bd385ec3"
format_stamp: "Formatted at 2026-09-27 22:45:59 on dist-test-slave-pp80"
server_key: "129360de7a26beb7dc3fabc80844c468"
server_key_iv: "25e984a258be61006889a6612d2bcb9a"
server_key_version: "encryptionkey@0"
I20260927 22:45:59.996603 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "cd3039483a1b4d22b05c6b14bd385ec3"
format_stamp: "Formatted at 2026-09-27 22:45:59 on dist-test-slave-pp80"
server_key: "129360de7a26beb7dc3fabc80844c468"
server_key_iv: "25e984a258be61006889a6612d2bcb9a"
server_key_version: "encryptionkey@0"
I20260927 22:46:00.002379 22762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:46:00.006251 22777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:00.007829 22762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:46:00.008057 22762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "cd3039483a1b4d22b05c6b14bd385ec3"
format_stamp: "Formatted at 2026-09-27 22:45:59 on dist-test-slave-pp80"
server_key: "129360de7a26beb7dc3fabc80844c468"
server_key_iv: "25e984a258be61006889a6612d2bcb9a"
server_key_version: "encryptionkey@0"
I20260927 22:46:00.008316 22762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:00.021718 22762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:00.026626 22762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:00.026939 22762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:00.046610 22762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46197
I20260927 22:46:00.046636 22828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46197 every 8 connection(s)
I20260927 22:46:00.048142 22762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:00.052623 22829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:00.054220 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22762
I20260927 22:46:00.054415 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:00.054940 1237 external_mini_cluster.cc:1428] Setting key 38b94af4500c949df61581e2226eee42
I20260927 22:46:00.067308 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3: Bootstrap starting.
I20260927 22:46:00.068944 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:00.056922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:55345 (local address 127.0.0.1:46197)
0927 22:46:00.057762 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:00.057774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:00.057823 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:46:00.059457 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:00.059481 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:00.059932 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:00.060363 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:00.060911 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:00.062217 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:00.064184 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:00.065003 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:00.065146 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:00.066477 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:00.066484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:00.066500 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:00.067060 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:00.067145 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:00.067171 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:00.067295 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:00.067676 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 22:46:00.067882 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":404,"threads_started":1}
I20260927 22:46:00.070683 22829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:00.071931 22829 log.cc:824] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:00.073021 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:00.075006 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3: No bootstrap required, opened a new log
I20260927 22:46:00.085019 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:00.077725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:37909 (local address 127.0.0.1:46197)
0927 22:46:00.077945 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:00.077951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:00.078304 (+ 353us) server_negotiation.cc:407] Connection header received
0927 22:46:00.078503 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:00.078507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:00.078574 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:00.078670 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:00.079308 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:00.080176 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:00.082140 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:00.082628 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:00.082732 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:00.084007 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:00.084012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:00.084015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:00.084033 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:00.084121 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:00.084143 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:00.084285 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:00.084518 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:46:00.084612 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:46:00.085341 22829 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46197 } }
I20260927 22:46:00.085896 22829 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:00.086027 22829 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3039483a1b4d22b05c6b14bd385ec3, State: Initialized, Role: FOLLOWER
I20260927 22:46:00.086809 22829 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46197 } }
I20260927 22:46:00.087072 22829 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:00.087358 22829 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:00.087628 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:00.090210 22829 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46197 } }
I20260927 22:46:00.090879 22829 leader_election.cc:302] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd3039483a1b4d22b05c6b14bd385ec3; no voters:
I20260927 22:46:00.091583 22829 leader_election.cc:288] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:00.091725 22839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:00.092509 22839 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [term 1 LEADER]: Becoming Leader. State: Replica: cd3039483a1b4d22b05c6b14bd385ec3, State: Running, Role: LEADER
I20260927 22:46:00.093146 22839 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46197 } }
I20260927 22:46:00.094014 22829 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:00.096271 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3039483a1b4d22b05c6b14bd385ec3. Latest consensus state: current_term: 1 leader_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46197 } } }
I20260927 22:46:00.096700 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:00.096586 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3039483a1b4d22b05c6b14bd385ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46197 } } }
I20260927 22:46:00.096864 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:00.098822 22847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:00.104400 22847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:00.114323 22847 catalog_manager.cc:1347] Generated new cluster ID: 3ad99e9d92914175b49c2c2de44826ec
I20260927 22:46:00.114455 22847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:00.129900 22847 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:00.131287 22847 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:00.140399 22847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd3039483a1b4d22b05c6b14bd385ec3: Generated new TSK 0
I20260927 22:46:00.141273 22847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:00.151288 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22762
W20260927 22:46:00.152456 22824 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.206:55345
W20260927 22:46:00.153249 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22762: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:46:00.153393 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22762
W20260927 22:46:00.170428 22747 connection.cc:535] client connection to 127.0.0.1:46197 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46197): Connection reset by peer (error 104)
2026-09-27T22:46:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:00 dist-test-slave-pp80 krb5kdc[22860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:00 dist-test-slave-pp80 krb5kdc[22860](info): set up 2 sockets
Sep 27 22:46:00 dist-test-slave-pp80 krb5kdc[22860](info): commencing operation
krb5kdc: starting...
W20260927 22:46:01.276669 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:01 dist-test-slave-pp80 krb5kdc[22860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:46:01.548804 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:01.919911 22873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:01.920322 22873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:01.920507 22873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:01.928098 22873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:01.928229 22873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:01.928349 22873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:01.928416 22873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:01.928462 22873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:01.939391 22873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:01.940840 22873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:01.942159 22873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:01.954804 22881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:01.954703 22878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:01.954838 22879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:01.955829 22873 server_base.cc:1034] running on GCE node
I20260927 22:46:01.956840 22873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:01.958446 22873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:01.959694 22873 hybrid_clock.cc:648] HybridClock initialized: now 1790549161959632 us; error 59 us; skew 500 ppm
I20260927 22:46:01.960068 22873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:01 dist-test-slave-pp80 krb5kdc[22860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:01.963490 22873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:46:01.965152 22873 webserver.cc:460] Webserver started at http://127.0.0.1:33319/ using document root <none> and password file <none>
I20260927 22:46:01.965888 22873 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:01.966054 22873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:01.966389 22873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:01.969743 22873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "deffde8c594c47c297267c9cf8ca268e"
format_stamp: "Formatted at 2026-09-27 22:46:01 on dist-test-slave-pp80"
server_key: "fb5e273295b7e9b5971fdafad9b7b11b"
server_key_iv: "e8f5dc2ca6c2ce24314aef072670e0d6"
server_key_version: "encryptionkey@0"
I20260927 22:46:01.970654 22873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "deffde8c594c47c297267c9cf8ca268e"
format_stamp: "Formatted at 2026-09-27 22:46:01 on dist-test-slave-pp80"
server_key: "fb5e273295b7e9b5971fdafad9b7b11b"
server_key_iv: "e8f5dc2ca6c2ce24314aef072670e0d6"
server_key_version: "encryptionkey@0"
I20260927 22:46:01.976374 22873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:46:01.980041 22888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:01.981490 22873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:46:01.981693 22873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "deffde8c594c47c297267c9cf8ca268e"
format_stamp: "Formatted at 2026-09-27 22:46:01 on dist-test-slave-pp80"
server_key: "fb5e273295b7e9b5971fdafad9b7b11b"
server_key_iv: "e8f5dc2ca6c2ce24314aef072670e0d6"
server_key_version: "encryptionkey@0"
I20260927 22:46:01.981905 22873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:01.987994 22873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:01.992772 22873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:01.993072 22873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:02.012217 22873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39417
I20260927 22:46:02.012315 22939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39417 every 8 connection(s)
I20260927 22:46:02.013743 22873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:02.015300 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22873
I20260927 22:46:02.015475 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:02.016040 1237 external_mini_cluster.cc:1428] Setting key d1740d18bf9dc39fbd35f0d0f39d9b31
I20260927 22:46:02.018602 22940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:02.024359 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:02.017937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:53831 (local address 127.0.0.1:39417)
0927 22:46:02.018784 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:02.018797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:02.018838 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:46:02.020667 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:02.020705 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:02.021111 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:02.021467 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:02.022525 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:02.022531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:02.022548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:02.022562 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:02.022639 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:02.022661 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:02.022838 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:02.022973 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 22:46:02.023276 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":384,"threads_started":1}
I20260927 22:46:02.028132 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:02.031232 22940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e: Bootstrap starting.
I20260927 22:46:02.035427 22940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:02.036911 22940 log.cc:824] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:02.038208 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:02.035625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.206:37893 (local address 127.0.0.1:39417)
0927 22:46:02.035832 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:02.035842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:02.036128 (+ 286us) server_negotiation.cc:407] Connection header received
0927 22:46:02.036332 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:02.036335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:02.036384 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:02.036462 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:02.037520 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:02.037525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:02.037528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:02.037543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:02.037588 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:02.037609 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:02.037724 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:02.037862 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:46:02.037925 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:46:02.040426 22940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e: No bootstrap required, opened a new log
I20260927 22:46:02.048198 22940 raft_consensus.cc:348] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deffde8c594c47c297267c9cf8ca268e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260927 22:46:02.048954 22940 raft_consensus.cc:374] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:02.049119 22940 raft_consensus.cc:729] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deffde8c594c47c297267c9cf8ca268e, State: Initialized, Role: FOLLOWER
I20260927 22:46:02.049716 22940 consensus_queue.cc:260] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deffde8c594c47c297267c9cf8ca268e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260927 22:46:02.049950 22940 raft_consensus.cc:388] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:02.050113 22940 raft_consensus.cc:482] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:02.050294 22940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:02.052578 22940 raft_consensus.cc:504] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deffde8c594c47c297267c9cf8ca268e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260927 22:46:02.053135 22940 leader_election.cc:302] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deffde8c594c47c297267c9cf8ca268e; no voters:
I20260927 22:46:02.053783 22940 leader_election.cc:288] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:02.053922 22950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:02.054653 22950 raft_consensus.cc:686] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [term 1 LEADER]: Becoming Leader. State: Replica: deffde8c594c47c297267c9cf8ca268e, State: Running, Role: LEADER
I20260927 22:46:02.055207 22950 consensus_queue.cc:237] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deffde8c594c47c297267c9cf8ca268e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260927 22:46:02.056151 22940 sys_catalog.cc:564] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:02.058234 22952 sys_catalog.cc:455] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [sys.catalog]: SysCatalogTable state changed. Reason: New leader deffde8c594c47c297267c9cf8ca268e. Latest consensus state: current_term: 1 leader_uuid: "deffde8c594c47c297267c9cf8ca268e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deffde8c594c47c297267c9cf8ca268e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } } }
I20260927 22:46:02.058223 22951 sys_catalog.cc:455] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deffde8c594c47c297267c9cf8ca268e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deffde8c594c47c297267c9cf8ca268e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } } }
I20260927 22:46:02.058687 22952 sys_catalog.cc:458] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:02.058682 22951 sys_catalog.cc:458] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:02.060459 22957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:02.065385 22957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:02.075079 22957 catalog_manager.cc:1347] Generated new cluster ID: d6b6f18a4892429c9042380ff5b3f98c
I20260927 22:46:02.075242 22957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:02.101572 22957 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:02.102061 22957 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:02.110707 22957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deffde8c594c47c297267c9cf8ca268e: Generated new TSK 0
I20260927 22:46:02.111599 22957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:02.117753 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22873
W20260927 22:46:02.119159 22935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.206:53831
W20260927 22:46:02.119959 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:46:02.120225 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22873
2026-09-27T22:46:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:02 dist-test-slave-pp80 krb5kdc[22971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:02 dist-test-slave-pp80 krb5kdc[22971](info): set up 2 sockets
Sep 27 22:46:02 dist-test-slave-pp80 krb5kdc[22971](info): commencing operation
krb5kdc: starting...
W20260927 22:46:03.253345 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:03 dist-test-slave-pp80 krb5kdc[22971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:03.524780 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37013
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:03.879027 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:03.879364 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:03.879537 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:03.887177 22984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:03.887313 22984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:03.887409 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:03.887508 22984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:03.887609 22984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:03.898222 22984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:37013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:37013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:03.899701 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:03.901216 22984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:03.914348 22992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:03.914448 22990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:03.914393 22989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:03.915735 22984 server_base.cc:1034] running on GCE node
I20260927 22:46:03.916985 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:03.918627 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:03.919860 22984 hybrid_clock.cc:648] HybridClock initialized: now 1790549163919763 us; error 101 us; skew 500 ppm
I20260927 22:46:03.920392 22984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:03 dist-test-slave-pp80 krb5kdc[22971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:03.923931 22984 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:03.925817 22984 webserver.cc:460] Webserver started at http://127.1.53.126:45985/ using document root <none> and password file <none>
I20260927 22:46:03.926589 22984 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:03.926746 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:03.927073 22984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:03.930276 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "135a928d1a2e40bca8f79a1d08bdd466"
format_stamp: "Formatted at 2026-09-27 22:46:03 on dist-test-slave-pp80"
server_key: "09eb35f0f3ccc3f2be6c52c1986a51ba"
server_key_iv: "26c55347cfba6f19bc7db9be107ddc0f"
server_key_version: "encryptionkey@0"
I20260927 22:46:03.931107 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "135a928d1a2e40bca8f79a1d08bdd466"
format_stamp: "Formatted at 2026-09-27 22:46:03 on dist-test-slave-pp80"
server_key: "09eb35f0f3ccc3f2be6c52c1986a51ba"
server_key_iv: "26c55347cfba6f19bc7db9be107ddc0f"
server_key_version: "encryptionkey@0"
I20260927 22:46:03.936874 22984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:46:03.940667 22999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:03.942273 22984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:46:03.942476 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "135a928d1a2e40bca8f79a1d08bdd466"
format_stamp: "Formatted at 2026-09-27 22:46:03 on dist-test-slave-pp80"
server_key: "09eb35f0f3ccc3f2be6c52c1986a51ba"
server_key_iv: "26c55347cfba6f19bc7db9be107ddc0f"
server_key_version: "encryptionkey@0"
I20260927 22:46:03.942698 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:03.962791 22984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:03.967293 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:03.967593 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:03.987087 22984 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:37013
I20260927 22:46:03.987135 23050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:37013 every 8 connection(s)
I20260927 22:46:03.988580 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:03.992480 23051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:03.994553 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 22984
I20260927 22:46:03.994736 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:03.995254 1237 external_mini_cluster.cc:1428] Setting key 23c11fdad9e6e9d8944678ebb2407b90
I20260927 22:46:04.007285 23051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466: Bootstrap starting.
Sep 27 22:46:04 dist-test-slave-pp80 krb5kdc[22971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:46:04.011445 23051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:04.012957 23051 log.cc:824] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:04.016597 23051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466: No bootstrap required, opened a new log
I20260927 22:46:04.020816 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:03.997152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55911 (local address 127.1.53.126:37013)
0927 22:46:03.998014 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:03.998026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:03.998069 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:46:03.999662 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:03.999688 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:04.000105 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:04.000500 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:04.002264 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:04.003616 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:04.005470 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:04.006238 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:04.006377 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:04.011438 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:04.011450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:04.011472 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:04.011530 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:04.015584 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:04.016595 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:04.016604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:04.016614 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:04.016822 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:04.017294 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:04.017299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:04.017302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:04.018060 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:04.018383 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:04.018908 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 22:46:04.019290 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":420,"threads_started":1}
I20260927 22:46:04.025010 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:04.026505 23051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a928d1a2e40bca8f79a1d08bdd466" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37013 } }
I20260927 22:46:04.026969 23051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:04.027074 23051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135a928d1a2e40bca8f79a1d08bdd466, State: Initialized, Role: FOLLOWER
I20260927 22:46:04.027745 23051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a928d1a2e40bca8f79a1d08bdd466" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37013 } }
I20260927 22:46:04.028035 23051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:04.028221 23051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:04.028438 23051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:04.031612 23051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a928d1a2e40bca8f79a1d08bdd466" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37013 } }
I20260927 22:46:04.032160 23051 leader_election.cc:302] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135a928d1a2e40bca8f79a1d08bdd466; no voters:
I20260927 22:46:04.033007 23051 leader_election.cc:288] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:04.033109 23061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:04.034574 23061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [term 1 LEADER]: Becoming Leader. State: Replica: 135a928d1a2e40bca8f79a1d08bdd466, State: Running, Role: LEADER
I20260927 22:46:04.035166 23061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a928d1a2e40bca8f79a1d08bdd466" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37013 } }
I20260927 22:46:04.035548 23051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:04.038228 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135a928d1a2e40bca8f79a1d08bdd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a928d1a2e40bca8f79a1d08bdd466" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37013 } } }
I20260927 22:46:04.038645 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:04.040354 23062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135a928d1a2e40bca8f79a1d08bdd466. Latest consensus state: current_term: 1 leader_uuid: "135a928d1a2e40bca8f79a1d08bdd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135a928d1a2e40bca8f79a1d08bdd466" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 37013 } } }
I20260927 22:46:04.040723 23062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:04.049003 23067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:04.050642 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:04.030166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43019 (local address 127.1.53.126:37013)
0927 22:46:04.030364 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:04.030370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:04.030900 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:46:04.031036 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:04.031041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:04.031118 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:04.031227 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:04.032787 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:04.034051 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:04.035756 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:04.036224 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:04.036437 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:04.041126 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:04.041136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:04.041144 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:04.041200 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:04.046076 (+ 4876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:04.047444 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:04.047453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:04.047460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:04.047687 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:04.048343 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:04.048354 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:04.048359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:04.048967 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:04.049324 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:04.049840 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 22:46:04.050099 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:46:04.054381 23067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:04.063705 23067 catalog_manager.cc:1347] Generated new cluster ID: c62341d227794e3bb4c33fa805eb5340
I20260927 22:46:04.063829 23067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:04.079263 23067 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:04.081248 23067 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:04.090854 23067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135a928d1a2e40bca8f79a1d08bdd466: Generated new TSK 0
I20260927 22:46:04.091869 23067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:04.102008 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 22984
I20260927 22:46:04.103753 23046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:04.196486 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 22984
W20260927 22:46:04.214419 22969 connection.cc:535] client connection to 127.1.53.126:37013 recv error: Network error: failed to read from TLS socket (remote: 127.1.53.126:37013): Connection reset by peer (error 104)
2026-09-27T22:46:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:04 dist-test-slave-pp80 krb5kdc[23083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:04 dist-test-slave-pp80 krb5kdc[23083](info): set up 2 sockets
Sep 27 22:46:04 dist-test-slave-pp80 krb5kdc[23083](info): commencing operation
krb5kdc: starting...
W20260927 22:46:05.313196 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:05 dist-test-slave-pp80 krb5kdc[23083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:05.580382 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45733
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:05.928063 23096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:05.928522 23096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:05.928692 23096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:05.936292 23096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:05.936461 23096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:05.936560 23096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:05.936679 23096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:05.936775 23096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:05.947237 23096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:45733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:05.948712 23096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:05.950112 23096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:05.962512 23104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:05.962451 23102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:05.962548 23101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:05.963169 23096 server_base.cc:1034] running on GCE node
I20260927 22:46:05.963943 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:05.965481 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:05.966698 23096 hybrid_clock.cc:648] HybridClock initialized: now 1790549165966636 us; error 54 us; skew 500 ppm
I20260927 22:46:05.967031 23096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:05 dist-test-slave-pp80 krb5kdc[23083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:05.970281 23096 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:05.971851 23096 webserver.cc:460] Webserver started at http://127.1.53.126:44757/ using document root <none> and password file <none>
I20260927 22:46:05.972640 23096 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:05.972770 23096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:05.973088 23096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:05.976209 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "d215e4e1eea642c6a69220acb0c6a516"
format_stamp: "Formatted at 2026-09-27 22:46:05 on dist-test-slave-pp80"
server_key: "9aff139244237c4fb2eb23e3d6b42026"
server_key_iv: "ccfe6df55fb5a0241e9a697793df330c"
server_key_version: "encryptionkey@0"
I20260927 22:46:05.977089 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "d215e4e1eea642c6a69220acb0c6a516"
format_stamp: "Formatted at 2026-09-27 22:46:05 on dist-test-slave-pp80"
server_key: "9aff139244237c4fb2eb23e3d6b42026"
server_key_iv: "ccfe6df55fb5a0241e9a697793df330c"
server_key_version: "encryptionkey@0"
I20260927 22:46:05.982596 23096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:46:05.986475 23111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:05.987973 23096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:46:05.988293 23096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "d215e4e1eea642c6a69220acb0c6a516"
format_stamp: "Formatted at 2026-09-27 22:46:05 on dist-test-slave-pp80"
server_key: "9aff139244237c4fb2eb23e3d6b42026"
server_key_iv: "ccfe6df55fb5a0241e9a697793df330c"
server_key_version: "encryptionkey@0"
I20260927 22:46:05.988528 23096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:06.001915 23096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:06.006458 23096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:06.006780 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:06.025893 23096 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:45733
I20260927 22:46:06.025928 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:45733 every 8 connection(s)
I20260927 22:46:06.027429 23096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:06.031162 23163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:06.037313 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23096
I20260927 22:46:06.037524 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:06.037937 1237 external_mini_cluster.cc:1428] Setting key b0d539b86e09566598c109c9fc9e0a0c
I20260927 22:46:06.042388 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516: Bootstrap starting.
I20260927 22:46:06.045729 23163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:06.046797 23163 log.cc:824] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:06.050535 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516: No bootstrap required, opened a new log
I20260927 22:46:06.052413 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:06.039796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60391 (local address 127.1.53.126:45733)
0927 22:46:06.040720 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:06.040733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:06.040782 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:46:06.042568 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:06.042608 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:06.043092 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:06.043499 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:06.044101 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:06.045475 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:06.046902 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:06.048223 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:06.048475 (+ 252us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:06.049827 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:06.049836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:06.049851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:06.050473 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:06.050562 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:06.050589 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:06.050770 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:06.051126 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 22:46:06.051302 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":475,"threads_started":1}
I20260927 22:46:06.056866 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:06.059614 23163 raft_consensus.cc:348] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d215e4e1eea642c6a69220acb0c6a516" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45733 } }
I20260927 22:46:06.060150 23163 raft_consensus.cc:374] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:06.060307 23163 raft_consensus.cc:729] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d215e4e1eea642c6a69220acb0c6a516, State: Initialized, Role: FOLLOWER
I20260927 22:46:06.060906 23163 consensus_queue.cc:260] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d215e4e1eea642c6a69220acb0c6a516" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45733 } }
I20260927 22:46:06.061172 23163 raft_consensus.cc:388] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:06.061333 23163 raft_consensus.cc:482] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:06.061538 23163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:06.064776 23163 raft_consensus.cc:504] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d215e4e1eea642c6a69220acb0c6a516" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45733 } }
I20260927 22:46:06.065330 23163 leader_election.cc:302] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d215e4e1eea642c6a69220acb0c6a516; no voters:
I20260927 22:46:06.066061 23163 leader_election.cc:288] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:06.066202 23173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:06.068133 23173 raft_consensus.cc:686] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [term 1 LEADER]: Becoming Leader. State: Replica: d215e4e1eea642c6a69220acb0c6a516, State: Running, Role: LEADER
I20260927 22:46:06.068763 23163 sys_catalog.cc:564] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:06.068837 23173 consensus_queue.cc:237] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d215e4e1eea642c6a69220acb0c6a516" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45733 } }
I20260927 22:46:06.069434 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:06.061413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56143 (local address 127.1.53.126:45733)
0927 22:46:06.061644 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:06.061652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:06.062103 (+ 451us) server_negotiation.cc:407] Connection header received
0927 22:46:06.062237 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:06.062247 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:06.062334 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:06.062445 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:06.063021 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:06.064188 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:06.065624 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:06.066453 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:06.067282 (+ 829us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:06.068082 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:06.068090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:06.068114 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:06.068140 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:06.068213 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:06.068267 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:06.068392 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:06.069048 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 22:46:06.069107 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:46:06.072638 23174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d215e4e1eea642c6a69220acb0c6a516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d215e4e1eea642c6a69220acb0c6a516" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45733 } } }
I20260927 22:46:06.072808 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d215e4e1eea642c6a69220acb0c6a516. Latest consensus state: current_term: 1 leader_uuid: "d215e4e1eea642c6a69220acb0c6a516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d215e4e1eea642c6a69220acb0c6a516" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 45733 } } }
I20260927 22:46:06.073072 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:06.073072 23174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:06.075129 23183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:06.080487 23183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:06.089022 23183 catalog_manager.cc:1347] Generated new cluster ID: 9b513d343780416f87aebd62913eacd3
I20260927 22:46:06.089154 23183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:06.115495 23183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:06.117133 23183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:06.125080 23183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d215e4e1eea642c6a69220acb0c6a516: Generated new TSK 0
I20260927 22:46:06.126276 23183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:06.141273 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23096
W20260927 22:46:06.142495 23158 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60391
W20260927 22:46:06.143319 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23096: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:46:06.143472 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23096
W20260927 22:46:06.161168 23081 connection.cc:535] client connection to 127.1.53.126:45733 recv error: Network error: failed to read from TLS socket (remote: 127.1.53.126:45733): Connection reset by peer (error 104)
2026-09-27T22:46:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:06 dist-test-slave-pp80 krb5kdc[23194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:06 dist-test-slave-pp80 krb5kdc[23194](info): set up 2 sockets
Sep 27 22:46:06 dist-test-slave-pp80 krb5kdc[23194](info): commencing operation
krb5kdc: starting...
W20260927 22:46:07.284806 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:07 dist-test-slave-pp80 krb5kdc[23194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:07.555164 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:40991
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:07.899744 23207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:07.900063 23207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:07.900158 23207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:07.907629 23207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:07.907768 23207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:07.907814 23207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:07.907876 23207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:07.907920 23207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:07.918107 23207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:40991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:07.919425 23207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:07.920712 23207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:07.933589 23215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:07.933641 23213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:07.933547 23212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:07.934635 23207 server_base.cc:1034] running on GCE node
I20260927 22:46:07.935539 23207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:07.937220 23207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:07.938488 23207 hybrid_clock.cc:648] HybridClock initialized: now 1790549167938434 us; error 46 us; skew 500 ppm
I20260927 22:46:07.938889 23207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:07 dist-test-slave-pp80 krb5kdc[23194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:07.953548 23207 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:07.955339 23207 webserver.cc:460] Webserver started at http://127.1.53.126:41625/ using document root <none> and password file <none>
I20260927 22:46:07.956058 23207 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:07.956166 23207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:07.956514 23207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:07.959618 23207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "7df71efb93b24f5c976fb96a825286a2"
format_stamp: "Formatted at 2026-09-27 22:46:07 on dist-test-slave-pp80"
server_key: "fa9e5b3991dd6c6660baad25a157d1d3"
server_key_iv: "b7fca1edf88b0233201a0ad7300500ae"
server_key_version: "encryptionkey@0"
I20260927 22:46:07.960438 23207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "7df71efb93b24f5c976fb96a825286a2"
format_stamp: "Formatted at 2026-09-27 22:46:07 on dist-test-slave-pp80"
server_key: "fa9e5b3991dd6c6660baad25a157d1d3"
server_key_iv: "b7fca1edf88b0233201a0ad7300500ae"
server_key_version: "encryptionkey@0"
I20260927 22:46:07.965775 23207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:46:07.969367 23222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:07.970805 23207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:46:07.970999 23207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "7df71efb93b24f5c976fb96a825286a2"
format_stamp: "Formatted at 2026-09-27 22:46:07 on dist-test-slave-pp80"
server_key: "fa9e5b3991dd6c6660baad25a157d1d3"
server_key_iv: "b7fca1edf88b0233201a0ad7300500ae"
server_key_version: "encryptionkey@0"
I20260927 22:46:07.971215 23207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:07.977604 23207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:07.982324 23207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:07.982648 23207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:08.002379 23207 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:40991
I20260927 22:46:08.002424 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:40991 every 8 connection(s)
I20260927 22:46:08.003865 23207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:08.007838 23274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:08.011811 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23207
I20260927 22:46:08.012012 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:08.012603 1237 external_mini_cluster.cc:1428] Setting key d0b47113bbf7464c4a90870f8b7dfbf9
I20260927 22:46:08.020699 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:08.014586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56831 (local address 127.1.53.126:40991)
0927 22:46:08.015383 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:08.015395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:08.015437 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:46:08.017133 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:08.017156 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:08.017531 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:08.017950 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:08.019016 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:08.019022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:08.019035 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:08.019049 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:08.019102 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:08.019123 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:08.019350 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:08.019486 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:46:08.019686 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":364,"threads_started":1}
I20260927 22:46:08.021020 23274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2: Bootstrap starting.
I20260927 22:46:08.024685 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:08.024709 23274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:08.025740 23274 log.cc:824] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:08.028825 23274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2: No bootstrap required, opened a new log
I20260927 22:46:08.031775 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:08.029201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40451 (local address 127.1.53.126:40991)
0927 22:46:08.029405 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:08.029414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:08.029754 (+ 340us) server_negotiation.cc:407] Connection header received
0927 22:46:08.029960 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:08.029964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:08.030012 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:08.030132 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:08.031062 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:08.031068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:08.031072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:08.031089 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:08.031131 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:08.031147 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:08.031291 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:08.031414 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 22:46:08.031479 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:46:08.037191 23274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df71efb93b24f5c976fb96a825286a2" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40991 } }
I20260927 22:46:08.037693 23274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:08.037812 23274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df71efb93b24f5c976fb96a825286a2, State: Initialized, Role: FOLLOWER
I20260927 22:46:08.038456 23274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df71efb93b24f5c976fb96a825286a2" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40991 } }
I20260927 22:46:08.038705 23274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:08.038862 23274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:08.039032 23274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:08.041694 23274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df71efb93b24f5c976fb96a825286a2" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40991 } }
I20260927 22:46:08.042318 23274 leader_election.cc:302] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df71efb93b24f5c976fb96a825286a2; no voters:
I20260927 22:46:08.042928 23274 leader_election.cc:288] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:08.043138 23284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:08.043825 23284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [term 1 LEADER]: Becoming Leader. State: Replica: 7df71efb93b24f5c976fb96a825286a2, State: Running, Role: LEADER
I20260927 22:46:08.044425 23284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df71efb93b24f5c976fb96a825286a2" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40991 } }
I20260927 22:46:08.045168 23274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:08.047530 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df71efb93b24f5c976fb96a825286a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df71efb93b24f5c976fb96a825286a2" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40991 } } }
I20260927 22:46:08.047917 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:08.049199 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df71efb93b24f5c976fb96a825286a2. Latest consensus state: current_term: 1 leader_uuid: "7df71efb93b24f5c976fb96a825286a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df71efb93b24f5c976fb96a825286a2" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 40991 } } }
I20260927 22:46:08.049583 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:08.058147 23292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:08.062808 23292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:08.071915 23292 catalog_manager.cc:1347] Generated new cluster ID: b477e7ba6cd442a9be53f7fcb22afc1b
I20260927 22:46:08.072067 23292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:08.086464 23292 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:08.087034 23292 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:08.104852 23292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df71efb93b24f5c976fb96a825286a2: Generated new TSK 0
I20260927 22:46:08.106077 23292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:08.113294 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23207
W20260927 22:46:08.114501 23269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56831
W20260927 22:46:08.115350 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:46:08.115520 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23207
2026-09-27T22:46:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:08 dist-test-slave-pp80 krb5kdc[23305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:08 dist-test-slave-pp80 krb5kdc[23305](info): set up 2 sockets
Sep 27 22:46:08 dist-test-slave-pp80 krb5kdc[23305](info): commencing operation
krb5kdc: starting...
W20260927 22:46:09.228832 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:09 dist-test-slave-pp80 krb5kdc[23305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:09.500272 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:43485
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:43485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:09.841063 23318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:09.841375 23318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:09.841565 23318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:09.849229 23318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:09.849349 23318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:09.849470 23318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:09.849568 23318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:09.849654 23318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:09.859929 23318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:43485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:09.861363 23318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:09.862691 23318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:09.874953 23323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:09.874979 23324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:09.875036 23326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:09.875964 23318 server_base.cc:1034] running on GCE node
I20260927 22:46:09.876817 23318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:09.878381 23318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:09.879637 23318 hybrid_clock.cc:648] HybridClock initialized: now 1790549169879575 us; error 45 us; skew 500 ppm
I20260927 22:46:09.879999 23318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:09 dist-test-slave-pp80 krb5kdc[23305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:09.883365 23318 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:09.885079 23318 webserver.cc:460] Webserver started at http://127.1.53.126:35295/ using document root <none> and password file <none>
I20260927 22:46:09.885859 23318 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:09.886034 23318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:09.886377 23318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:09.889808 23318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "c29cba261ae548a09794bc93183f00a9"
format_stamp: "Formatted at 2026-09-27 22:46:09 on dist-test-slave-pp80"
server_key: "ceced53286cd42031ecb28d9a3aa648c"
server_key_iv: "6c3be7a5702b6126dfbc1d0961f5c0b3"
server_key_version: "encryptionkey@0"
I20260927 22:46:09.890694 23318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "c29cba261ae548a09794bc93183f00a9"
format_stamp: "Formatted at 2026-09-27 22:46:09 on dist-test-slave-pp80"
server_key: "ceced53286cd42031ecb28d9a3aa648c"
server_key_iv: "6c3be7a5702b6126dfbc1d0961f5c0b3"
server_key_version: "encryptionkey@0"
I20260927 22:46:09.896898 23318 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 22:46:09.900964 23333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:09.902452 23318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:46:09.902681 23318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "c29cba261ae548a09794bc93183f00a9"
format_stamp: "Formatted at 2026-09-27 22:46:09 on dist-test-slave-pp80"
server_key: "ceced53286cd42031ecb28d9a3aa648c"
server_key_iv: "6c3be7a5702b6126dfbc1d0961f5c0b3"
server_key_version: "encryptionkey@0"
I20260927 22:46:09.902910 23318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:09.929484 23318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:09.934451 23318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:09.934778 23318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:09.953985 23318 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:43485
I20260927 22:46:09.954006 23384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:43485 every 8 connection(s)
I20260927 22:46:09.955443 23318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:09.957140 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23318
I20260927 22:46:09.957314 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:09.957849 1237 external_mini_cluster.cc:1428] Setting key e4e4ff18ace7682934e102f389804ea6
I20260927 22:46:09.960284 23385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:09.971452 23385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9: Bootstrap starting.
Sep 27 22:46:09 dist-test-slave-pp80 krb5kdc[23305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:46:09.975127 23385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:09.976321 23385 log.cc:824] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:09.979864 23385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9: No bootstrap required, opened a new log
I20260927 22:46:09.981057 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:09.959688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46279 (local address 127.1.53.126:43485)
0927 22:46:09.960587 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:09.960605 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:09.960648 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:46:09.962293 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:09.962315 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:09.962727 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:09.963153 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:09.964659 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:09.965998 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:09.967394 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:09.968124 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:09.968354 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:09.972865 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:09.972877 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:09.972892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:09.972941 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:09.976125 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:09.977001 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:09.977011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:09.977024 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:09.977215 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:09.977652 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:09.977655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:09.977657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:09.978495 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:09.978802 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:09.979182 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 22:46:09.979619 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":354,"threads_started":1}
I20260927 22:46:09.984988 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:09.988775 23385 raft_consensus.cc:348] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29cba261ae548a09794bc93183f00a9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43485 } }
I20260927 22:46:09.989202 23385 raft_consensus.cc:374] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:09.989357 23385 raft_consensus.cc:729] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29cba261ae548a09794bc93183f00a9, State: Initialized, Role: FOLLOWER
I20260927 22:46:09.990064 23385 consensus_queue.cc:260] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29cba261ae548a09794bc93183f00a9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43485 } }
I20260927 22:46:09.990368 23385 raft_consensus.cc:388] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:09.990535 23385 raft_consensus.cc:482] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:09.990723 23385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:09.993512 23385 raft_consensus.cc:504] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29cba261ae548a09794bc93183f00a9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43485 } }
I20260927 22:46:09.994032 23385 leader_election.cc:302] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c29cba261ae548a09794bc93183f00a9; no voters:
I20260927 22:46:09.994724 23385 leader_election.cc:288] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:09.994844 23395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:09.996191 23395 raft_consensus.cc:686] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [term 1 LEADER]: Becoming Leader. State: Replica: c29cba261ae548a09794bc93183f00a9, State: Running, Role: LEADER
I20260927 22:46:09.996903 23395 consensus_queue.cc:237] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29cba261ae548a09794bc93183f00a9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43485 } }
I20260927 22:46:09.997236 23385 sys_catalog.cc:564] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:09.999652 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c29cba261ae548a09794bc93183f00a9. Latest consensus state: current_term: 1 leader_uuid: "c29cba261ae548a09794bc93183f00a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29cba261ae548a09794bc93183f00a9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43485 } } }
I20260927 22:46:10.000105 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:10.000685 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c29cba261ae548a09794bc93183f00a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29cba261ae548a09794bc93183f00a9" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 43485 } } }
I20260927 22:46:10.001060 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:10.011855 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:09.989497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58925 (local address 127.1.53.126:43485)
0927 22:46:09.989688 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:09.989694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:09.990170 (+ 476us) server_negotiation.cc:407] Connection header received
0927 22:46:09.990306 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:09.990311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:09.990412 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:09.990532 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:09.992091 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:09.993226 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:09.994680 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:09.995490 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:09.995613 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:09.999187 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:09.999197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:09.999205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:09.999259 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:10.004071 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:10.005667 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:10.005676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:10.005682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:10.006152 (+ 470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:10.007362 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:10.007372 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:10.007378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:10.008366 (+ 988us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:10.010744 (+ 2378us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:10.010969 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:46:10.011301 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:46:10.021252 23402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:10.026770 23402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:10.036327 23402 catalog_manager.cc:1347] Generated new cluster ID: 516b817cee914b9b93963be105d346e5
I20260927 22:46:10.036458 23402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:10.054486 23402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:10.056535 23402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:10.075723 23402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c29cba261ae548a09794bc93183f00a9: Generated new TSK 0
I20260927 22:46:10.076934 23402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:10.092411 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23318
I20260927 22:46:10.093653 23380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:10.188300 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23318
2026-09-27T22:46:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:10 dist-test-slave-pp80 krb5kdc[23417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:10 dist-test-slave-pp80 krb5kdc[23417](info): set up 2 sockets
Sep 27 22:46:10 dist-test-slave-pp80 krb5kdc[23417](info): commencing operation
krb5kdc: starting...
W20260927 22:46:11.304715 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:11 dist-test-slave-pp80 krb5kdc[23417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:11.571756 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42109
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:42109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:11.931964 23430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:11.932346 23430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:11.932523 23430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:11.939945 23430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:11.940121 23430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:11.940256 23430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:11.940349 23430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:11.940438 23430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:11.950624 23430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:42109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:42109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:11.951970 23430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:11.953367 23430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:11.965509 23436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:11.965575 23438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:11.965559 23435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:11.966364 23430 server_base.cc:1034] running on GCE node
I20260927 22:46:11.967224 23430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:11.968817 23430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:11.970049 23430 hybrid_clock.cc:648] HybridClock initialized: now 1790549171970000 us; error 49 us; skew 500 ppm
I20260927 22:46:11.970408 23430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:11 dist-test-slave-pp80 krb5kdc[23417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549171, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:11.973583 23430 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:11.975083 23430 webserver.cc:460] Webserver started at http://127.1.53.126:34981/ using document root <none> and password file <none>
I20260927 22:46:11.975845 23430 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:11.975996 23430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:11.976408 23430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:11.979516 23430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "0c3374cd6fd74a88ada5af38fabb3a23"
format_stamp: "Formatted at 2026-09-27 22:46:11 on dist-test-slave-pp80"
server_key: "44d4c7d042dce68f07225d427b38ba70"
server_key_iv: "95c6cbff5661dafbfcf7397b639e661d"
server_key_version: "encryptionkey@0"
I20260927 22:46:11.980401 23430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "0c3374cd6fd74a88ada5af38fabb3a23"
format_stamp: "Formatted at 2026-09-27 22:46:11 on dist-test-slave-pp80"
server_key: "44d4c7d042dce68f07225d427b38ba70"
server_key_iv: "95c6cbff5661dafbfcf7397b639e661d"
server_key_version: "encryptionkey@0"
I20260927 22:46:11.985912 23430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:46:11.989718 23445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:11.991111 23430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:46:11.991315 23430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "0c3374cd6fd74a88ada5af38fabb3a23"
format_stamp: "Formatted at 2026-09-27 22:46:11 on dist-test-slave-pp80"
server_key: "44d4c7d042dce68f07225d427b38ba70"
server_key_iv: "95c6cbff5661dafbfcf7397b639e661d"
server_key_version: "encryptionkey@0"
I20260927 22:46:11.991529 23430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:12.021750 23430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:12.026125 23430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:12.026427 23430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:12.045544 23430 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:42109
I20260927 22:46:12.045579 23496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:42109 every 8 connection(s)
I20260927 22:46:12.047026 23430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:12.048033 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23430
I20260927 22:46:12.048278 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:12.048713 1237 external_mini_cluster.cc:1428] Setting key 6efeedfa68f6cca52d0877685112905a
I20260927 22:46:12.051800 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:12.062281 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:12.050626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37355 (local address 127.1.53.126:42109)
0927 22:46:12.051448 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:12.051460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:12.051503 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:46:12.053334 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:12.053357 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:12.053801 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:12.054234 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:12.054835 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:12.056154 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:12.057527 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:12.058227 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:12.058440 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:12.059704 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:12.059712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:12.059727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:12.060401 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:12.060488 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:12.060519 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:12.060657 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:12.061025 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:46:12.061297 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":359,"threads_started":1}
I20260927 22:46:12.063489 23497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23: Bootstrap starting.
I20260927 22:46:12.066232 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:12.067054 23497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:12.068089 23497 log.cc:824] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:12.071280 23497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23: No bootstrap required, opened a new log
I20260927 22:46:12.077714 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:12.070686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41339 (local address 127.1.53.126:42109)
0927 22:46:12.070879 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:12.070886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:12.071371 (+ 485us) server_negotiation.cc:407] Connection header received
0927 22:46:12.071579 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:12.071586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:12.071661 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:12.071792 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:12.072340 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:12.073423 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:12.074774 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:12.075511 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:12.075644 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:12.076831 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:12.076837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:12.076842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:12.076865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:12.076911 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:12.076936 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:12.077038 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:12.077244 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:46:12.077341 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:46:12.080508 23497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42109 } }
I20260927 22:46:12.080900 23497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:12.081003 23497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3374cd6fd74a88ada5af38fabb3a23, State: Initialized, Role: FOLLOWER
I20260927 22:46:12.081648 23497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42109 } }
I20260927 22:46:12.081887 23497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:12.082042 23497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:12.082228 23497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:12.084578 23497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42109 } }
I20260927 22:46:12.085112 23497 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3374cd6fd74a88ada5af38fabb3a23; no voters:
I20260927 22:46:12.085723 23497 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:12.085930 23507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:12.086851 23507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [term 1 LEADER]: Becoming Leader. State: Replica: 0c3374cd6fd74a88ada5af38fabb3a23, State: Running, Role: LEADER
I20260927 22:46:12.087427 23507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42109 } }
I20260927 22:46:12.088001 23497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:12.090574 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42109 } } }
I20260927 22:46:12.091071 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:12.091848 23509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3374cd6fd74a88ada5af38fabb3a23. Latest consensus state: current_term: 1 leader_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3374cd6fd74a88ada5af38fabb3a23" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 42109 } } }
I20260927 22:46:12.092125 23509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:12.093271 23516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:12.099822 23516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:12.111249 23516 catalog_manager.cc:1347] Generated new cluster ID: 013491a771a94359af5a72815f169071
I20260927 22:46:12.111395 23516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:12.125676 23516 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:12.127835 23516 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:12.139708 23516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3374cd6fd74a88ada5af38fabb3a23: Generated new TSK 0
I20260927 22:46:12.140676 23516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:12.148461 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23430
W20260927 22:46:12.149544 23492 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37355
W20260927 22:46:12.150244 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:46:12.150362 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23430
2026-09-27T22:46:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:12 dist-test-slave-pp80 krb5kdc[23528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:12 dist-test-slave-pp80 krb5kdc[23528](info): set up 2 sockets
Sep 27 22:46:12 dist-test-slave-pp80 krb5kdc[23528](info): commencing operation
krb5kdc: starting...
W20260927 22:46:13.285923 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:13 dist-test-slave-pp80 krb5kdc[23528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:13.551884 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:34079
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:13.891822 23541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:13.892155 23541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:13.892350 23541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:13.899857 23541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:13.899979 23541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:13.900075 23541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:13.900194 23541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:13.900312 23541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:13.910357 23541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:34079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:13.911729 23541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:13.913081 23541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:13.925309 23546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:13.925388 23549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:13.925410 23547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:13.926443 23541 server_base.cc:1034] running on GCE node
I20260927 22:46:13.927219 23541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:13.928702 23541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:13.929912 23541 hybrid_clock.cc:648] HybridClock initialized: now 1790549173929824 us; error 76 us; skew 500 ppm
I20260927 22:46:13.930305 23541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:13 dist-test-slave-pp80 krb5kdc[23528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549173, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:13.939774 23541 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:13.941474 23541 webserver.cc:460] Webserver started at http://127.1.53.126:38491/ using document root <none> and password file <none>
I20260927 22:46:13.942247 23541 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:13.942391 23541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:13.942723 23541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:13.945880 23541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "abff5ba5fce3436287bfda30b0f3772c"
format_stamp: "Formatted at 2026-09-27 22:46:13 on dist-test-slave-pp80"
server_key: "2c11be17fa50ecee9fc4968abf19453e"
server_key_iv: "96d196c77e97cf5469635851292487bb"
server_key_version: "encryptionkey@0"
I20260927 22:46:13.946753 23541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "abff5ba5fce3436287bfda30b0f3772c"
format_stamp: "Formatted at 2026-09-27 22:46:13 on dist-test-slave-pp80"
server_key: "2c11be17fa50ecee9fc4968abf19453e"
server_key_iv: "96d196c77e97cf5469635851292487bb"
server_key_version: "encryptionkey@0"
I20260927 22:46:13.952215 23541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260927 22:46:13.955979 23556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:13.957621 23541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:46:13.957823 23541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "abff5ba5fce3436287bfda30b0f3772c"
format_stamp: "Formatted at 2026-09-27 22:46:13 on dist-test-slave-pp80"
server_key: "2c11be17fa50ecee9fc4968abf19453e"
server_key_iv: "96d196c77e97cf5469635851292487bb"
server_key_version: "encryptionkey@0"
I20260927 22:46:13.958043 23541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:13.963960 23541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:13.968411 23541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:13.968719 23541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:13.987455 23541 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:34079
I20260927 22:46:13.987512 23607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:34079 every 8 connection(s)
I20260927 22:46:13.988997 23541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:13.992849 23608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:13.998047 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23541
I20260927 22:46:13.998282 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:13.998754 1237 external_mini_cluster.cc:1428] Setting key 063b943dd07ac6c4b5eebca095336f14
I20260927 22:46:14.005057 23608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c: Bootstrap starting.
I20260927 22:46:14.008313 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:14.000861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58977 (local address 127.1.53.126:34079)
0927 22:46:14.001776 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:14.001794 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:14.001856 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:46:14.004196 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:14.004228 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:14.004728 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:14.005184 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:14.006262 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:14.006273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:14.006289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:14.006316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:14.006386 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:14.006420 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:14.006638 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:14.006790 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:46:14.007025 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":409,"threads_started":1}
I20260927 22:46:14.009274 23608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:14.010392 23608 log.cc:824] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:14.012391 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:14.013441 23608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c: No bootstrap required, opened a new log
I20260927 22:46:14.019237 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:14.016668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43027 (local address 127.1.53.126:34079)
0927 22:46:14.016892 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:14.016900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:14.017257 (+ 357us) server_negotiation.cc:407] Connection header received
0927 22:46:14.017463 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:14.017474 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:14.017520 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:14.017623 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:14.018545 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:14.018551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:14.018555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:46:14.018578 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:14.018628 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:46:14.018643 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:14.018789 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:14.018899 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 22:46:14.018960 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:46:14.022526 23608 raft_consensus.cc:348] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff5ba5fce3436287bfda30b0f3772c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34079 } }
I20260927 22:46:14.022935 23608 raft_consensus.cc:374] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:46:14.023052 23608 raft_consensus.cc:729] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abff5ba5fce3436287bfda30b0f3772c, State: Initialized, Role: FOLLOWER
I20260927 22:46:14.023779 23608 consensus_queue.cc:260] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff5ba5fce3436287bfda30b0f3772c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34079 } }
I20260927 22:46:14.024082 23608 raft_consensus.cc:388] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:46:14.024237 23608 raft_consensus.cc:482] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:46:14.024474 23608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:14.026700 23608 raft_consensus.cc:504] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff5ba5fce3436287bfda30b0f3772c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34079 } }
I20260927 22:46:14.027213 23608 leader_election.cc:302] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abff5ba5fce3436287bfda30b0f3772c; no voters:
I20260927 22:46:14.027876 23608 leader_election.cc:288] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:46:14.028023 23618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:14.030011 23618 raft_consensus.cc:686] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [term 1 LEADER]: Becoming Leader. State: Replica: abff5ba5fce3436287bfda30b0f3772c, State: Running, Role: LEADER
I20260927 22:46:14.030094 23608 sys_catalog.cc:564] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:14.030537 23618 consensus_queue.cc:237] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff5ba5fce3436287bfda30b0f3772c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34079 } }
I20260927 22:46:14.033461 23620 sys_catalog.cc:455] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abff5ba5fce3436287bfda30b0f3772c. Latest consensus state: current_term: 1 leader_uuid: "abff5ba5fce3436287bfda30b0f3772c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff5ba5fce3436287bfda30b0f3772c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34079 } } }
I20260927 22:46:14.033854 23620 sys_catalog.cc:458] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:14.034505 23619 sys_catalog.cc:455] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abff5ba5fce3436287bfda30b0f3772c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abff5ba5fce3436287bfda30b0f3772c" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 34079 } } }
I20260927 22:46:14.034844 23619 sys_catalog.cc:458] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:14.036080 23626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:14.041603 23626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:14.050681 23626 catalog_manager.cc:1347] Generated new cluster ID: 53613c3b2d91425b805fbcf90c116897
I20260927 22:46:14.050807 23626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:14.085520 23626 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:14.086030 23626 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:14.102186 23626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abff5ba5fce3436287bfda30b0f3772c: Generated new TSK 0
I20260927 22:46:14.103103 23626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:14.113567 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23541
W20260927 22:46:14.115046 23603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58977
W20260927 22:46:14.115772 1237 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:46:14.115926 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23541
2026-09-27T22:46:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1965 ms)
[----------] 18 tests from AuthTokenIssuingTest (36064 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:14 dist-test-slave-pp80 krb5kdc[23639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:14 dist-test-slave-pp80 krb5kdc[23639](info): set up 2 sockets
Sep 27 22:46:14 dist-test-slave-pp80 krb5kdc[23639](info): commencing operation
krb5kdc: starting...
W20260927 22:46:15.257012 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:15 dist-test-slave-pp80 krb5kdc[23639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:15.528441 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:38673
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:38673,127.1.53.125:39735,127.1.53.124:36027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:15.869628 23652 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:46:15.870010 23652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:15.870117 23652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:15.870249 23652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:15.877736 23652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:15.877841 23652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:15.877938 23652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:15.878023 23652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:15.878131 23652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:15.888664 23652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:38673,127.1.53.125:39735,127.1.53.124:36027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:15.890022 23652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:15.891386 23652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:15.903635 23657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:15.903759 23658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:15.903734 23660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:15.904912 23652 server_base.cc:1034] running on GCE node
I20260927 22:46:15.905833 23652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:15.907399 23652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:15.908650 23652 hybrid_clock.cc:648] HybridClock initialized: now 1790549175908583 us; error 46 us; skew 500 ppm
I20260927 22:46:15.909010 23652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:15 dist-test-slave-pp80 krb5kdc[23639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:15.912458 23652 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:15.914076 23652 webserver.cc:460] Webserver started at http://127.1.53.126:43681/ using document root <none> and password file <none>
I20260927 22:46:15.914811 23652 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:15.914947 23652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:15.915298 23652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:15.918478 23652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "2aed7d0151ce4e5a96485de545443eff"
format_stamp: "Formatted at 2026-09-27 22:46:15 on dist-test-slave-pp80"
server_key: "1fdd0415834707dad8503366389d93f4"
server_key_iv: "ad6c0c58dfcd74b74e9f19fd2b7d032e"
server_key_version: "encryptionkey@0"
I20260927 22:46:15.919318 23652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "2aed7d0151ce4e5a96485de545443eff"
format_stamp: "Formatted at 2026-09-27 22:46:15 on dist-test-slave-pp80"
server_key: "1fdd0415834707dad8503366389d93f4"
server_key_iv: "ad6c0c58dfcd74b74e9f19fd2b7d032e"
server_key_version: "encryptionkey@0"
I20260927 22:46:15.925431 23652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 22:46:15.929323 23667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:15.930805 23652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:46:15.931012 23652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "2aed7d0151ce4e5a96485de545443eff"
format_stamp: "Formatted at 2026-09-27 22:46:15 on dist-test-slave-pp80"
server_key: "1fdd0415834707dad8503366389d93f4"
server_key_iv: "ad6c0c58dfcd74b74e9f19fd2b7d032e"
server_key_version: "encryptionkey@0"
I20260927 22:46:15.931249 23652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:15.953598 23652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:15.958232 23652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:15.958518 23652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:15.977010 23652 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:38673
I20260927 22:46:15.977030 23718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:38673 every 8 connection(s)
I20260927 22:46:15.978427 23652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:15.982179 23719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:15.984875 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23652
I20260927 22:46:15.985154 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:15.985774 1237 external_mini_cluster.cc:1428] Setting key 35f72e3fa96d2df0f27a194c12b7b9de
I20260927 22:46:15.989156 23719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:16 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.125@KRBTEST.COM" created.
I20260927 22:46:16.015250 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:15.992460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:38673 (local address 127.0.0.1:47576)
0927 22:46:15.993387 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:15.993442 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:15.996207 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:15.997161 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:15.997175 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:15.997993 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:15.998504 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:15.998559 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.000284 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.000292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.002203 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.002221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.003164 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.003186 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.003414 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.004125 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:16.004164 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:16.007039 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:16.010800 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.010813 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.010827 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.011297 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.011721 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.011727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.011731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.011882 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.012748 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:16.012757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:16.012985 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 22:46:16.013240 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:46:16.013825 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":138,"thread_start_us":447,"threads_started":1}
I20260927 22:46:16.015281 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:15.992470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.1.53.126:38673)
0927 22:46:15.994203 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:15.994212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:15.994266 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:46:15.996656 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:15.996662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:15.996906 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:15.997098 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:15.998788 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.000083 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.002358 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.002911 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.003098 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.007310 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:16.007319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:16.007333 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:16.007368 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:16.010634 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.011442 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.011448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.011450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.011608 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.012011 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.012015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.012017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.012609 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:16.012824 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:16.013357 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 22:46:16.013708 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1519,"thread_start_us":1319,"threads_started":1}
I20260927 22:46:16.018560 23719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:46:16.019793 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.019357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55308)
0927 22:46:16.019563 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.019676 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 22:46:16.020128 23671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
I20260927 22:46:16.021806 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.021347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55316)
0927 22:46:16.021525 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.021619 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:46:16.022130 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39735: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab.
Entry for principal kudu/127.1.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab.
I20260927 22:46:16.047463 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:39735
--webserver_interface=127.1.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab
--principal=kudu/127.1.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:38673,127.1.53.125:39735,127.1.53.124:36027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:46:16.071525 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } attempt: 1
I20260927 22:46:16.072814 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.072338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55332)
0927 22:46:16.072555 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.072672 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:46:16.073869 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.073489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55342)
0927 22:46:16.073680 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.073747 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:46:16.074075 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39735: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
I20260927 22:46:16.142395 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } attempt: 2
I20260927 22:46:16.143653 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.143173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55344)
0927 22:46:16.143388 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.143507 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:46:16.144840 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.144448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55360)
0927 22:46:16.144624 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.144706 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:46:16.145104 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39735: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
I20260927 22:46:16.236361 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } attempt: 3
I20260927 22:46:16.237689 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.237247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55366)
0927 22:46:16.237451 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.237542 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:46:16.238862 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.238467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55376)
0927 22:46:16.238593 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.238694 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:46:16.239144 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39735: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
I20260927 22:46:16.382416 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } attempt: 4
I20260927 22:46:16.383731 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.383171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55378)
0927 22:46:16.383490 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.383577 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260927 22:46:16.384837 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.384472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55394)
0927 22:46:16.384631 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.384702 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:46:16.385120 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39735: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39735: connect: Connection refused (error 111)
W20260927 22:46:16.395241 23726 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:46:16.395596 23726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:16.395733 23726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:16.395855 23726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:16.403360 23726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:16.403461 23726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:16.403568 23726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:16.403643 23726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:16.403734 23726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:16.413759 23726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:38673,127.1.53.125:39735,127.1.53.124:36027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab
--principal=kudu/127.1.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:39735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:16.415113 23726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:16.416455 23726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:16.428594 23731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:16.428586 23732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:16.428898 23734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:16.429816 23726 server_base.cc:1034] running on GCE node
I20260927 22:46:16.430707 23726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:16.432354 23726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:16.433677 23726 hybrid_clock.cc:648] HybridClock initialized: now 1790549176433576 us; error 52 us; skew 500 ppm
I20260927 22:46:16.434073 23726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:16 dist-test-slave-pp80 krb5kdc[23639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:16.437668 23726 init.cc:377] Logged in from keytab as kudu/127.1.53.125@KRBTEST.COM (short username kudu)
I20260927 22:46:16.439239 23726 webserver.cc:460] Webserver started at http://127.1.53.125:45783/ using document root <none> and password file <none>
I20260927 22:46:16.439973 23726 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:16.440117 23726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:16.440502 23726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:16.443631 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data/instance:
uuid: "e24351f02ff24aa1b91954b62f6b56eb"
format_stamp: "Formatted at 2026-09-27 22:46:16 on dist-test-slave-pp80"
server_key: "fa875185a3c20d3755d46ea2a701d051"
server_key_iv: "0d7f61a313030e94d9b601476644df25"
server_key_version: "encryptionkey@0"
I20260927 22:46:16.444540 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal/instance:
uuid: "e24351f02ff24aa1b91954b62f6b56eb"
format_stamp: "Formatted at 2026-09-27 22:46:16 on dist-test-slave-pp80"
server_key: "fa875185a3c20d3755d46ea2a701d051"
server_key_iv: "0d7f61a313030e94d9b601476644df25"
server_key_version: "encryptionkey@0"
I20260927 22:46:16.450008 23726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:46:16.453773 23741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:16.455166 23726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:46:16.455348 23726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal
uuid: "e24351f02ff24aa1b91954b62f6b56eb"
format_stamp: "Formatted at 2026-09-27 22:46:16 on dist-test-slave-pp80"
server_key: "fa875185a3c20d3755d46ea2a701d051"
server_key_iv: "0d7f61a313030e94d9b601476644df25"
server_key_version: "encryptionkey@0"
I20260927 22:46:16.455591 23726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:16.467345 23726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:16.471853 23726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:16.472144 23726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:16.491266 23726 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.125:39735
I20260927 22:46:16.491291 23792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.125:39735 every 8 connection(s)
I20260927 22:46:16.492743 23726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
I20260927 22:46:16.493901 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23726
I20260927 22:46:16.494100 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-1/wal/instance
I20260927 22:46:16.494628 1237 external_mini_cluster.cc:1428] Setting key d0ad7baf89e8271d7ffe44888d2bfa7b
I20260927 22:46:16.497661 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:16.507342 23793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.124@KRBTEST.COM" created.
Sep 27 22:46:16 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:46:16.532634 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.510193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.1.53.126:38673)
0927 22:46:16.510452 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:16.510461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:16.511737 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 22:46:16.513787 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:16.513795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:16.513891 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:16.514015 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:16.515653 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.516886 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.518536 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.519272 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.519422 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.525306 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:16.525327 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:16.525336 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:16.525391 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:16.529092 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.529918 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.529926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.529933 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.530095 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.530529 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.530537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.530543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.530963 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:16.531129 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:16.531894 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 22:46:16.532174 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:46:16.533381 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.510664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:38673 (local address 127.0.0.1:47580)
0927 22:46:16.511537 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.511592 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:16.513492 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:16.514259 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:16.514268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:16.514865 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:16.515466 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.515483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.517041 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.517045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.518369 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.518382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.519499 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.519504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.519728 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.520488 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:16.520510 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:16.525014 (+ 4504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:16.529262 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.529269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.529285 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.529781 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.530225 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.530228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.530230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.530412 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.531107 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:16.531114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:16.531495 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 22:46:16.531738 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 22:46:16.532143 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260927 22:46:16.535403 23793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:16 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:46:16.550431 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.536394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55404)
0927 22:46:16.536564 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.536584 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:16.536723 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:16.537417 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:16.537422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:16.537871 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:16.537998 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.538006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.539356 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.539364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.540232 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.540295 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.541128 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.541133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.541308 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.541941 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:16.541957 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:16.544391 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:16.547789 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.547796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.547800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.548161 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.548548 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.548555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.548559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.548700 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.549365 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:16.549370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:16.549512 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 22:46:16.549667 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:46:16.549958 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260927 22:46:16.550468 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.536427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.1.53.125:39735)
0927 22:46:16.537011 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:16.537017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:16.537041 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:46:16.537082 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:16.537087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:16.537272 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:16.537428 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:16.538183 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.539228 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.540431 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.540982 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.541129 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.544598 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:16.544616 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:16.544632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:16.544665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:16.547649 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.548328 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.548335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.548337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.548445 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.548810 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.548821 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.548824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.549244 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:16.549462 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:16.549848 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:46:16.550058 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":274,"threads_started":1}
I20260927 22:46:16.553591 23793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:46:16.554890 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.554374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59070)
0927 22:46:16.554568 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.554680 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 22:46:16.555227 23742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.556923 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.556598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59078)
0927 22:46:16.556736 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.556814 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:46:16.557183 23793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab.
Entry for principal kudu/127.1.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab.
I20260927 22:46:16.559278 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:36027
--webserver_interface=127.1.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab
--principal=kudu/127.1.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:38673,127.1.53.125:39735,127.1.53.124:36027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:46:16.606531 23793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 1
I20260927 22:46:16.607847 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.607313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59092)
0927 22:46:16.607509 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.607693 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:46:16.608861 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.608515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59100)
0927 22:46:16.608694 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.608761 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:46:16.609040 23793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.677314 23793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 2
I20260927 22:46:16.678545 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.678084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59108)
0927 22:46:16.678292 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.678388 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 22:46:16.679611 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.679262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59118)
0927 22:46:16.679409 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.679489 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:46:16.679804 23793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.684365 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } attempt: 5
Sep 27 22:46:16 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:46:16.697777 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.685035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55412)
0927 22:46:16.685234 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.685255 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:16.685394 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:16.685760 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:16.685764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:16.686216 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:16.686391 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.686400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.687409 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.687414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.688349 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:16.688361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.689114 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.689119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:16.689334 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.689976 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:16.689994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:16.692303 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:16.695284 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.695290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.695294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.695698 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.696074 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:16.696079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:16.696082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:16.696221 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:16.696790 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:16.696795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:16.696959 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 22:46:16.697098 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:46:16.697398 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:46:16.697777 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.685129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.1.53.125:39735)
0927 22:46:16.685289 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:16.685297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:16.685319 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:46:16.685537 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:16.685541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:16.685619 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:16.685728 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:16.686542 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.687250 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.688497 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:16.688928 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:16.689069 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:16.692508 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:16.692513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:16.692516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:16.692550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:16.695138 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.695822 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.695829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.695833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.695959 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:16.696365 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:16.696370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:16.696374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:16.696671 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:16.696829 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:16.697223 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:46:16.697447 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:46:16.699039 23719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:46:16.700174 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.699691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59132)
0927 22:46:16.699884 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.699985 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 22:46:16.701146 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.700763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59144)
0927 22:46:16.700919 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.701020 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:46:16.701375 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.752627 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 1
I20260927 22:46:16.754079 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.753414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59150)
0927 22:46:16.753691 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.753833 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 22:46:16.755213 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.754755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59154)
0927 22:46:16.754949 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.755051 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:46:16.755465 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.771168 23793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 3
I20260927 22:46:16.772388 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.771850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59170)
0927 22:46:16.772045 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.772155 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:46:16.773552 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.773125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59176)
0927 22:46:16.773359 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.773438 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 22:46:16.773895 23793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.823721 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 2
I20260927 22:46:16.825047 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.824450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59186)
0927 22:46:16.824701 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.824838 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 22:46:16.826156 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.825723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59188)
0927 22:46:16.825896 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.826000 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:46:16.826385 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
W20260927 22:46:16.912212 23800 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:46:16.912685 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:16.912819 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:16.912947 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:46:16.917146 23793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 4
I20260927 22:46:16.918330 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.917839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59192)
0927 22:46:16.918075 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.918171 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:46:16.919355 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.919029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59196)
0927 22:46:16.919168 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.919251 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:46:16.919600 23793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
W20260927 22:46:16.921063 23800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:16.921152 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:16.921240 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:16.921312 23800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:16.921406 23800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:16.932024 23800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:38673,127.1.53.125:39735,127.1.53.124:36027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab
--principal=kudu/127.1.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:36027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:16.932613 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 3
I20260927 22:46:16.933707 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:16.933770 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.933296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59198)
0927 22:46:16.933519 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.933612 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:46:16.935177 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:16.934567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59200)
0927 22:46:16.934855 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:16.934955 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 22:46:16.935496 23719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:36027: Network error: Client connection negotiation failed: client connection to 127.1.53.124:36027: connect: Connection refused (error 111)
I20260927 22:46:16.935640 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:16.948122 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:16.947980 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:16.948093 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:16.948997 23800 server_base.cc:1034] running on GCE node
I20260927 22:46:16.949823 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:16.951324 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:16.952555 23800 hybrid_clock.cc:648] HybridClock initialized: now 1790549176952463 us; error 49 us; skew 500 ppm
I20260927 22:46:16.952925 23800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:16 dist-test-slave-pp80 krb5kdc[23639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:16.956914 23800 init.cc:377] Logged in from keytab as kudu/127.1.53.124@KRBTEST.COM (short username kudu)
I20260927 22:46:16.958474 23800 webserver.cc:460] Webserver started at http://127.1.53.124:37407/ using document root <none> and password file <none>
I20260927 22:46:16.959302 23800 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:16.959443 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:16.959774 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:16.962936 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data/instance:
uuid: "e4a25f09e1224700a0ba84e0d37ad358"
format_stamp: "Formatted at 2026-09-27 22:46:16 on dist-test-slave-pp80"
server_key: "cae051bfd61e341449120613b75efaab"
server_key_iv: "308e0b653eecf8db50ae5e83934ab933"
server_key_version: "encryptionkey@0"
I20260927 22:46:16.963764 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal/instance:
uuid: "e4a25f09e1224700a0ba84e0d37ad358"
format_stamp: "Formatted at 2026-09-27 22:46:16 on dist-test-slave-pp80"
server_key: "cae051bfd61e341449120613b75efaab"
server_key_iv: "308e0b653eecf8db50ae5e83934ab933"
server_key_version: "encryptionkey@0"
I20260927 22:46:16.969302 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 22:46:16.973230 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:16.974835 23800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:46:16.975046 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal
uuid: "e4a25f09e1224700a0ba84e0d37ad358"
format_stamp: "Formatted at 2026-09-27 22:46:16 on dist-test-slave-pp80"
server_key: "cae051bfd61e341449120613b75efaab"
server_key_iv: "308e0b653eecf8db50ae5e83934ab933"
server_key_version: "encryptionkey@0"
I20260927 22:46:16.975293 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:17.015012 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:17.019711 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:17.019999 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:17.039834 23800 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.124:36027
I20260927 22:46:17.039855 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.124:36027 every 8 connection(s)
I20260927 22:46:17.041304 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
I20260927 22:46:17.045252 23867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:17.046307 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 23800
I20260927 22:46:17.046499 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790548812741971-1237-0/minicluster-data/master-2/wal/instance
I20260927 22:46:17.046947 1237 external_mini_cluster.cc:1428] Setting key e0ca7b95fc341e3e63382c399d74d081
I20260927 22:46:17.055641 23867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:46:17.073576 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.049221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.1.53.124:36027)
0927 22:46:17.050072 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.050083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.050125 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:46:17.051809 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.051832 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.052343 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.052793 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.054716 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.056050 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.058103 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.059139 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.059352 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.064808 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.064818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.064835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.064889 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.068290 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.069207 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.069218 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.069227 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.069442 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.069878 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.069883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.069887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.070706 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.071108 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.071457 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.072111 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":413,"threads_started":1}
I20260927 22:46:17.076376 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.058907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:38673 (local address 127.0.0.1:47584)
0927 22:46:17.059968 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:17.060099 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:17.060334 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:17.060929 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:17.060938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:17.062042 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:17.062454 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.062471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.063603 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.063610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.064881 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.064893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.065706 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.065714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.065870 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.066489 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:17.066531 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:17.069089 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:17.073149 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.073159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.073165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.073654 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.074327 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.074332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.074335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.074511 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.075147 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:17.075156 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:17.075339 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 22:46:17.075542 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 22:46:17.075908 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":477,"threads_started":1}
I20260927 22:46:17.076382 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.058978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.1.53.126:38673)
0927 22:46:17.060569 (+ 1591us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.060574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.060590 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:46:17.060646 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.060650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.060708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.060795 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.062642 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.063445 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.065042 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.065502 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.065666 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.069313 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.069323 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.069331 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.069375 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.072985 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.073849 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.073855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.073859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.074034 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.074674 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.074679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.074681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.074954 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.075155 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.075658 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.075899 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":1240,"threads_started":1}
I20260927 22:46:17.077843 23867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:46:17.093385 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.078525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39735 (local address 127.0.0.1:55428)
0927 22:46:17.078715 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:17.078735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:17.078830 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:17.079218 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:17.079223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:17.079626 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:17.079760 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.079767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.080951 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.080957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.081997 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.082010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.082928 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.082934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.083152 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.083796 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:17.083824 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:17.086546 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:17.090445 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.090450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.090454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.090926 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.091455 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.091460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.091463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.091657 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.092376 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:17.092381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:17.092543 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 22:46:17.092698 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:46:17.093018 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:46:17.093434 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.076970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.1.53.126:38673)
0927 22:46:17.077435 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.077440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.077454 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:46:17.077491 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.077494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.077559 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.077667 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.079163 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.079902 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.081388 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.081972 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.082127 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.087560 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.087565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.087568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.087602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.090509 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.091294 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.091298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.091300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.091474 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.091932 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.091945 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.091947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.092286 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.092549 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.092877 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.093118 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260927 22:46:17.093459 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.078645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.1.53.125:39735)
0927 22:46:17.078813 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.078818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.078834 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:46:17.078945 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.078949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.079079 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.079195 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.079979 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.080781 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.082163 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.082748 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.082882 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.086820 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.086829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.086835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.086885 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.090269 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.091130 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.091137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.091140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.091332 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.091774 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.091779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.091783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.092174 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.092421 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.092826 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.093071 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:46:17.094736 23867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:46:17.110471 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.095433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59206)
0927 22:46:17.095624 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:17.095644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:17.095781 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:17.096165 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:17.096168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:17.096747 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:17.096964 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.096978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.098048 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.098052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.099206 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.099221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.100143 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.100152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.100411 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.101157 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:17.101182 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:17.103917 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:17.107680 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.107689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.107694 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.108099 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.108534 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.108541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.108544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.108693 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.109280 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:17.109286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:17.109446 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 22:46:17.109615 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 22:46:17.110012 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":26}
I20260927 22:46:17.110513 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.095655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.1.53.124:36027)
0927 22:46:17.095773 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.095777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.095792 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:46:17.095894 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.095898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.095989 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.096109 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.097141 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.097882 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.099353 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.099956 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.100056 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.104171 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.104176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.104180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.104210 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.107520 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.108228 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.108260 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.108267 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.108412 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.108806 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.108814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.108817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.109148 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.109306 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.109697 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.110174 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":35}
I20260927 22:46:17.111454 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.095365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.1.53.125:39735)
0927 22:46:17.095529 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.095535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.095966 (+ 431us) server_negotiation.cc:407] Connection header received
0927 22:46:17.096106 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.096111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.096176 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.096329 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.097871 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.098666 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.100215 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.100763 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.100897 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.105713 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.105719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.105723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.105759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.108729 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.109440 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.109446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.109450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.109595 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.109990 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.109995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.109998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.110277 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.110560 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.110861 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.111090 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:46:17.112749 23719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 4
I20260927 22:46:17.116448 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:17.117682 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: Bootstrap starting.
I20260927 22:46:17.121961 23867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:17.123178 23867 log.cc:824] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:17.119819 23694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2aed7d0151ce4e5a96485de545443eff"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47598
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549175, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:46:17.126950 23867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: No bootstrap required, opened a new log
I20260927 22:46:17.132594 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.116581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.1.53.124:36027)
0927 22:46:17.116790 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.116796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.116814 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:46:17.116867 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.116871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.116927 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.117045 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.117825 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.118576 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.120279 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.121031 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.121167 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.125574 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.125581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.125586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.125628 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.129400 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.130270 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.130276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.130279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.130446 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.130926 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.130931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.130934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.131347 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.131511 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.131930 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.132166 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:46:17.132617 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.113441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59212)
0927 22:46:17.113684 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:17.113706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:17.113821 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:17.117140 (+ 3319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:17.117148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:17.117526 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:17.117682 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.117691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.118745 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.118749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.120089 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.120104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.121217 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.121222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.121509 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.122280 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:17.122310 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:17.125348 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:17.129585 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.129591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.129595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.130142 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.130597 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.130604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.130610 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.130809 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.131483 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:17.131488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:17.131667 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:46:17.131810 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:46:17.132134 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:46:17.136758 23867 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.137131 23867 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a25f09e1224700a0ba84e0d37ad358, State: Initialized, Role: FOLLOWER
I20260927 22:46:17.137734 23719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: Bootstrap starting.
I20260927 22:46:17.137727 23867 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.138767 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.139158 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.140062 23867 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:17.141484 23719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:17.142697 23719 log.cc:824] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:17.146582 23719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: No bootstrap required, opened a new log
W20260927 22:46:17.152611 23886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:46:17.152786 23886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:46:17.154922 23719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.155398 23719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aed7d0151ce4e5a96485de545443eff, State: Initialized, Role: FOLLOWER
I20260927 22:46:17.155954 23719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.157423 23888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.158248 23888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.158551 23719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:17.158641 23694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:46:17.158856 23694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:17.149736 23800 master_runner.cc:186] Error getting master registration for 127.1.53.126:38673: OK, instance_id { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" instance_seqno: 1790549175971655 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:46:17.162766 23800 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:46:17.162794 23694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.167271 23842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aed7d0151ce4e5a96485de545443eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e4a25f09e1224700a0ba84e0d37ad358"
I20260927 22:46:17.167848 23842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:17.171891 23842 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aed7d0151ce4e5a96485de545443eff in term 1.
I20260927 22:46:17.172816 23668 leader_election.cc:302] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aed7d0151ce4e5a96485de545443eff, e4a25f09e1224700a0ba84e0d37ad358; no voters:
I20260927 22:46:17.178336 23694 leader_election.cc:288] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [CANDIDATE]: Term 1 election: Requested vote from peers e24351f02ff24aa1b91954b62f6b56eb (127.1.53.125:39735), e4a25f09e1224700a0ba84e0d37ad358 (127.1.53.124:36027)
W20260927 22:46:17.179386 23899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:46:17.179522 23899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:46:17.180361 23901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:17.173120 23652 master_runner.cc:186] Error getting master registration for 127.1.53.125:39735: OK, instance_id { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" instance_seqno: 1790549176485838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:46:17.187493 23652 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:46:17.197687 23901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [term 1 LEADER]: Becoming Leader. State: Replica: 2aed7d0151ce4e5a96485de545443eff, State: Running, Role: LEADER
I20260927 22:46:17.198449 23901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
W20260927 22:46:17.211632 23673 tablet.cc:2369] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:46:17.215327 23888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aed7d0151ce4e5a96485de545443eff. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.215715 23888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:17.218807 23793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } attempt: 5
I20260927 22:46:17.240624 23911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:46:17.243060 23911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:17.246104 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.205632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.1.53.124:36027)
0927 22:46:17.205844 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.205851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.213399 (+ 7548us) server_negotiation.cc:407] Connection header received
0927 22:46:17.213727 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.213733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.213838 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.213919 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.219820 (+ 5901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.220985 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.232385 (+ 11400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.233185 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.233300 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.237422 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.237431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.237439 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.237491 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.241961 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.243009 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.243016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.243020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.243298 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.243831 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.243837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.243842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.244328 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.244655 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.245302 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.245662 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:46:17 dist-test-slave-pp80 krb5kdc[23639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549176, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:46:17.258435 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.193475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.1.53.125:39735)
0927 22:46:17.193690 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.193698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.204803 (+ 11105us) server_negotiation.cc:407] Connection header received
0927 22:46:17.206654 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.206661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.206761 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.206848 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.214106 (+ 7258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.215527 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.229018 (+ 13491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.229845 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.229969 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.239139 (+ 9170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.239148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.239154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.239215 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.244522 (+ 5307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.253182 (+ 8660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.253191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.253195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.253365 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.254153 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.254160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.254164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.254608 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.254859 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.257695 (+ 2836us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.257990 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:46:17.261656 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.194697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.1.53.126:38673)
0927 22:46:17.194936 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.194943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.210696 (+ 15753us) server_negotiation.cc:407] Connection header received
0927 22:46:17.211283 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.211291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.211394 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.211487 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.223776 (+ 12289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.225136 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.232379 (+ 7243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.233184 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.233303 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.245598 (+ 12295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.245607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.245612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.245658 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.250196 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.255414 (+ 5218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.255422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.255429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.255622 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.256208 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.256215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.256222 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.256698 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.257046 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.260888 (+ 3842us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.261205 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:46:17.263556 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.219461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:36027 (local address 127.0.0.1:59230)
0927 22:46:17.219879 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:17.219901 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:17.220021 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:17.239589 (+ 19568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:17.239599 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:17.240705 (+ 1106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:17.240946 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.240961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.245884 (+ 4923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.245891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.247342 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:17.247360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.248545 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.248551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:17.248780 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.249401 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:17.249426 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:17.252285 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:17.260476 (+ 8191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.260482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.260486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.261067 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.261624 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:17.261630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:17.261633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:17.261800 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:17.262514 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:17.262519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:17.262686 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 22:46:17.262840 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:46:17.263158 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260927 22:46:17.265388 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.220270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.1.53.124:36027)
0927 22:46:17.239066 (+ 18796us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.239072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.239100 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:46:17.239161 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.239165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.239259 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.239745 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.244354 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.245708 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.247561 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.248225 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.252477 (+ 4252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.252769 (+ 292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:17.252776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:17.252781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:17.252816 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:17.256952 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.261260 (+ 4308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.261267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.261270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.261467 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:17.261950 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:17.261955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:17.261958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:17.262349 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:17.264544 (+ 2195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.264686 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.264930 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18622,"thread_start_us":19077,"threads_started":1}
I20260927 22:46:17.271471 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: Bootstrap starting.
W20260927 22:46:17.272370 23821 tablet.cc:2369] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:46:17.276481 23793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:17.277884 23793 log.cc:824] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:17.284318 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: No bootstrap required, opened a new log
I20260927 22:46:17.294410 23842 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [term 1 FOLLOWER]: Refusing update from remote peer 2aed7d0151ce4e5a96485de545443eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:46:17.295545 23888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:46:17.300760 23793 raft_consensus.cc:348] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.301339 23793 raft_consensus.cc:729] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24351f02ff24aa1b91954b62f6b56eb, State: Initialized, Role: FOLLOWER
I20260927 22:46:17.301929 23793 consensus_queue.cc:260] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } }
I20260927 22:46:17.309052 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.311486 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.311326 23793 sys_catalog.cc:564] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:17.311794 23767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:17.311404 23768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2aed7d0151ce4e5a96485de545443eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e24351f02ff24aa1b91954b62f6b56eb"
I20260927 22:46:17.316113 23767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [term 1 FOLLOWER]: Refusing update from remote peer 2aed7d0151ce4e5a96485de545443eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:46:17.317298 23888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:46:17.320305 23768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2aed7d0151ce4e5a96485de545443eff in term 1.
I20260927 22:46:17.330962 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aed7d0151ce4e5a96485de545443eff. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.331314 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.334370 23913 mvcc.cc:205] Tried to move back new op lower bound from 7334089430182952960 to 7334089429813272576. Current Snapshot: MvccSnapshot[applied={T|T < 7334089430182952960}]
I20260927 22:46:17.335230 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.335567 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:17.338161 23911 catalog_manager.cc:1347] Generated new cluster ID: d2952cabcb5d44c89ec87a303872bea2
I20260927 22:46:17.338271 23911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:17.338899 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.339198 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.353010 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aed7d0151ce4e5a96485de545443eff. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.353374 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.359789 23902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.360280 23902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:17.363705 23911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:17.365697 23911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:17.374459 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2aed7d0151ce4e5a96485de545443eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aed7d0151ce4e5a96485de545443eff" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 38673 } } peers { permanent_uuid: "e24351f02ff24aa1b91954b62f6b56eb" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39735 } } peers { permanent_uuid: "e4a25f09e1224700a0ba84e0d37ad358" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 36027 } } }
I20260927 22:46:17.374752 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:17.395239 23911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aed7d0151ce4e5a96485de545443eff: Generated new TSK 0
I20260927 22:46:17.396485 23911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:46:17.421727 23933 catalog_manager.cc:1259] Loaded cluster ID: d2952cabcb5d44c89ec87a303872bea2
I20260927 22:46:17.421911 23933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: loading cluster ID for follower catalog manager: success
I20260927 22:46:17.425801 23933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: acquiring CA information for follower catalog manager: success
I20260927 22:46:17.439134 23933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e24351f02ff24aa1b91954b62f6b56eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:46:17.545338 23798 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.535391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.1.53.125:39735)
0927 22:46:17.535627 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.535634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.541538 (+ 5904us) server_negotiation.cc:407] Connection header received
0927 22:46:17.543509 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.543518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.543626 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.543704 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.545077 (+ 1373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55456: BlockingRecv error: recv got EOF from 127.0.0.1:55456 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 22:46:17.547647 23872 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.526667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.1.53.126:38673)
0927 22:46:17.526914 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.526925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.541043 (+ 14118us) server_negotiation.cc:407] Connection header received
0927 22:46:17.546013 (+ 4970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.546020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.546120 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.546199 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.547385 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47610: BlockingRecv error: recv got EOF from 127.0.0.1:47610 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260927 22:46:17.551707 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.542244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.1.53.124:36027)
0927 22:46:17.542426 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.542436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.549152 (+ 6716us) server_negotiation.cc:407] Connection header received
0927 22:46:17.550274 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.550280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.550389 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.550461 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.551456 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59236: BlockingRecv error: recv got EOF from 127.0.0.1:59236 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:46:17.606395 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.591178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.1.53.125:39735)
0927 22:46:17.591369 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.591376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.598279 (+ 6903us) server_negotiation.cc:407] Connection header received
0927 22:46:17.598819 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.598826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.599030 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.599186 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.599939 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.601144 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.603180 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.603902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.604013 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.604843 (+ 830us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.605526 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.605653 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.605891 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.606013 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:46:17.675420 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.661377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.1.53.124:36027)
0927 22:46:17.661578 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.661585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.672753 (+ 11168us) server_negotiation.cc:407] Connection header received
0927 22:46:17.673646 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.673653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.673796 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.673889 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.675097 (+ 1208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59248: BlockingRecv error: recv got EOF from 127.0.0.1:59248 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:46:17.726167 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.711311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.1.53.125:39735)
0927 22:46:17.711529 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.711535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.718691 (+ 7156us) server_negotiation.cc:407] Connection header received
0927 22:46:17.719308 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.719316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.719450 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.719525 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.720368 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.721554 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.723686 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.724445 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.724548 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.725207 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.725381 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.725450 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.725699 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.725797 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:46:17.782013 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.766250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.1.53.124:36027)
0927 22:46:17.766494 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.766501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.779890 (+ 13389us) server_negotiation.cc:407] Connection header received
0927 22:46:17.780265 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.780271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.780378 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.780461 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.781772 (+ 1311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59252: BlockingRecv error: recv got EOF from 127.0.0.1:59252 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:46:17.836229 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.826370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.1.53.125:39735)
0927 22:46:17.826595 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.826601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.828915 (+ 2314us) server_negotiation.cc:407] Connection header received
0927 22:46:17.829153 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.829161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.829264 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.829381 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.829898 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.831240 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.833357 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.834077 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.834232 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.835152 (+ 920us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.835316 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.835422 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.835688 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.835849 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 22:46:17.846493 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.844121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.1.53.124:36027)
0927 22:46:17.844346 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.844353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.844846 (+ 493us) server_negotiation.cc:407] Connection header received
0927 22:46:17.845046 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.845058 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.845157 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.845271 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.846225 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59266: BlockingRecv error: recv got EOF from 127.0.0.1:59266 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:46:17.863618 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.854764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.1.53.125:39735)
0927 22:46:17.854997 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.855005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.856854 (+ 1849us) server_negotiation.cc:407] Connection header received
0927 22:46:17.857155 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.857162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.857250 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.857358 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.857862 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.859154 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.861356 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.862160 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.862315 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.862827 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.862959 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.863062 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.863244 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.863328 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 22:46:17.873113 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.871062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.1.53.124:36027)
0927 22:46:17.871273 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.871279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.871299 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:46:17.871536 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.871543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.871645 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.871723 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.872849 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59272: BlockingRecv error: recv got EOF from 127.0.0.1:59272 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 22:46:17.897454 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.889503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.1.53.125:39735)
0927 22:46:17.889681 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.889687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.890212 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:46:17.890518 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.890526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.890633 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.890746 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.891274 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.892570 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.894976 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.895704 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.895849 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.896545 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.896712 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.896810 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.897009 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.897108 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:46:17.912799 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.910592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.1.53.124:36027)
0927 22:46:17.910775 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.910779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.911273 (+ 494us) server_negotiation.cc:407] Connection header received
0927 22:46:17.911434 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.911440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.911558 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.911681 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.912552 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59288: BlockingRecv error: recv got EOF from 127.0.0.1:59288 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:46:17.938762 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.932534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.1.53.125:39735)
0927 22:46:17.932743 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.932749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.933220 (+ 471us) server_negotiation.cc:407] Connection header received
0927 22:46:17.933318 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.933324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.933440 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.933558 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.933935 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.935211 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.936734 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.937434 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.937583 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.937989 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.938154 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.938255 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.938371 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.938462 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 22:46:17.947183 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.945028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.1.53.124:36027)
0927 22:46:17.945293 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.945299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.945632 (+ 333us) server_negotiation.cc:407] Connection header received
0927 22:46:17.945841 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.945845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.945917 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.946011 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.946906 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59298: BlockingRecv error: recv got EOF from 127.0.0.1:59298 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:46:17.990763 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:17.983602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.1.53.125:39735)
0927 22:46:17.983811 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.983817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.984455 (+ 638us) server_negotiation.cc:407] Connection header received
0927 22:46:17.984694 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.984700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.984828 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.984924 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:17.985406 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.986557 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.988538 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:17.989309 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:17.989443 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:17.989910 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:17.990071 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:17.990165 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:17.990336 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:46:17.990420 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:46:17.999676 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:17.997862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.1.53.124:36027)
0927 22:46:17.998057 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:17.998063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:17.998425 (+ 362us) server_negotiation.cc:407] Connection header received
0927 22:46:17.998567 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:17.998571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:17.998641 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:17.998727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:17.999428 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59314: BlockingRecv error: recv got EOF from 127.0.0.1:59314 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:46:18.072175 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:18.067153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.1.53.125:39735)
0927 22:46:18.067397 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.067404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.067597 (+ 193us) server_negotiation.cc:407] Connection header received
0927 22:46:18.067766 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.067770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.067839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.067919 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:18.068358 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.069136 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.070410 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.070843 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.070940 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:18.071400 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:18.071533 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:18.071614 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:18.071744 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 22:46:18.071813 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260927 22:46:18.080276 23912 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:18.078143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.1.53.124:36027)
0927 22:46:18.078424 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.078433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.078700 (+ 267us) server_negotiation.cc:407] Connection header received
0927 22:46:18.078925 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.078931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.079079 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.079208 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:18.079980 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59316: BlockingRecv error: recv got EOF from 127.0.0.1:59316 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:46:18.154773 23886 catalog_manager.cc:1259] Loaded cluster ID: d2952cabcb5d44c89ec87a303872bea2
I20260927 22:46:18.154879 23886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: loading cluster ID for follower catalog manager: success
I20260927 22:46:18.157421 23886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: acquiring CA information for follower catalog manager: success
I20260927 22:46:18.159333 23886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4a25f09e1224700a0ba84e0d37ad358: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:46:18.219720 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:18.213881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.1.53.125:39735)
0927 22:46:18.214787 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.214794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.214882 (+ 88us) server_negotiation.cc:407] Connection header received
0927 22:46:18.215135 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.215139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.215216 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.215298 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:18.215753 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.216556 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.217913 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.218375 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.218515 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:18.218961 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:18.219114 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:18.219217 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:18.219365 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:46:18.219446 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753}
I20260927 22:46:18.236603 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:18.230528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.1.53.124:36027)
0927 22:46:18.230769 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.230777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.231154 (+ 377us) server_negotiation.cc:407] Connection header received
0927 22:46:18.231397 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.231407 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.231574 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.231702 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:18.232132 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.232967 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.234303 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.234728 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.234865 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:18.235405 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:18.235962 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:18.236144 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:18.236194 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:46:18.236308 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:46:18.251088 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:18.242640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.1.53.126:38673)
0927 22:46:18.243379 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.243387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.244077 (+ 690us) server_negotiation.cc:407] Connection header received
0927 22:46:18.244314 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.244320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.244567 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.244756 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:18.245466 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.246401 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.248107 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.248597 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.248733 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:18.249943 (+ 1210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:18.250358 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:18.250474 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:18.250610 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:46:18.250717 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":506,"threads_started":1}
I20260927 22:46:18.251168 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:18.244002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.1.53.124:36027)
0927 22:46:18.244219 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.244228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.244801 (+ 573us) server_negotiation.cc:407] Connection header received
0927 22:46:18.245095 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.245099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.245197 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.245288 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:18.246114 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.247053 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.249229 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.249655 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.249764 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:18.250466 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:18.250618 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:18.250676 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:18.250887 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 22:46:18.250965 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:46:18.251255 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:18.242722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.1.53.125:39735)
0927 22:46:18.242969 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:18.242978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:18.244000 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 22:46:18.245065 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:18.245072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:18.245188 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:18.245286 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:18.245955 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.246860 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.249217 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:18.249668 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:18.249764 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:18.250427 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:18.250614 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:18.250712 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:18.250907 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:46:18.250986 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:46:18.257771 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23652
I20260927 22:46:18.259387 23714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:18.375592 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23652
W20260927 22:46:18.397272 23818 connection.cc:535] server connection from 127.0.0.1:59212 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:46:18.397693 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23726
I20260927 22:46:18.398820 23788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:18.508117 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23726
I20260927 22:46:18.528777 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 23800
I20260927 22:46:18.529997 23862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:18.639261 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 23800
2026-09-27T22:46:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4528 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:46:18 dist-test-slave-pp80 krb5kdc[24047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:46:18 dist-test-slave-pp80 krb5kdc[24047](info): set up 2 sockets
Sep 27 22:46:18 dist-test-slave-pp80 krb5kdc[24047](info): commencing operation
krb5kdc: starting...
W20260927 22:46:19.760931 1237 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/test-user.keytab.
Sep 27 22:46:19 dist-test-slave-pp80 krb5kdc[24047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
Entry for principal kudu/127.1.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab.
I20260927 22:46:20.036635 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46337
--webserver_interface=127.1.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46337,127.1.53.125:39561,127.1.53.124:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:46:20.388212 24060 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:46:20.388624 24060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:20.388751 24060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:20.388888 24060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:20.396479 24060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:20.396576 24060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:20.396697 24060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:20.396787 24060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:20.396883 24060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:20.407289 24060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46337,127.1.53.125:39561,127.1.53.124:38923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.126.keytab
--principal=kudu/127.1.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.126:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:20.408749 24060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:20.410116 24060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:20.422668 24065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:20.422827 24068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:20.422616 24066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:20.423727 24060 server_base.cc:1034] running on GCE node
I20260927 22:46:20.424659 24060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:20.426200 24060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:20.427476 24060 hybrid_clock.cc:648] HybridClock initialized: now 1790549180427419 us; error 39 us; skew 500 ppm
I20260927 22:46:20.427866 24060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:20 dist-test-slave-pp80 krb5kdc[24047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:20.439514 24060 init.cc:377] Logged in from keytab as kudu/127.1.53.126@KRBTEST.COM (short username kudu)
I20260927 22:46:20.441238 24060 webserver.cc:460] Webserver started at http://127.1.53.126:41103/ using document root <none> and password file <none>
I20260927 22:46:20.442013 24060 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:20.442165 24060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:20.442502 24060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:20.445703 24060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data/instance:
uuid: "c443365fc8bc4966b0507d3151187249"
format_stamp: "Formatted at 2026-09-27 22:46:20 on dist-test-slave-pp80"
server_key: "e5d2595c57d19493cbe8c1205166acb4"
server_key_iv: "48419cad387b2cc16bfc66e723097f85"
server_key_version: "encryptionkey@0"
I20260927 22:46:20.446542 24060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal/instance:
uuid: "c443365fc8bc4966b0507d3151187249"
format_stamp: "Formatted at 2026-09-27 22:46:20 on dist-test-slave-pp80"
server_key: "e5d2595c57d19493cbe8c1205166acb4"
server_key_iv: "48419cad387b2cc16bfc66e723097f85"
server_key_version: "encryptionkey@0"
I20260927 22:46:20.452278 24060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:46:20.455821 24075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:20.457403 24060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:46:20.457602 24060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal
uuid: "c443365fc8bc4966b0507d3151187249"
format_stamp: "Formatted at 2026-09-27 22:46:20 on dist-test-slave-pp80"
server_key: "e5d2595c57d19493cbe8c1205166acb4"
server_key_iv: "48419cad387b2cc16bfc66e723097f85"
server_key_version: "encryptionkey@0"
I20260927 22:46:20.457844 24060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:20.483846 24060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:20.488260 24060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:20.488560 24060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:20.507974 24060 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.126:46337
I20260927 22:46:20.507985 24126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.126:46337 every 8 connection(s)
I20260927 22:46:20.509426 24060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/data/info.pb
I20260927 22:46:20.513243 24127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:20.514405 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 24060
I20260927 22:46:20.514645 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-0/wal/instance
I20260927 22:46:20.515344 1237 external_mini_cluster.cc:1428] Setting key cff873767dfbbeb9e1c2eb0a7b4c869e
I20260927 22:46:20.520624 24127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:20 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.125@KRBTEST.COM" created.
I20260927 22:46:20.547924 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.524368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46337 (local address 127.0.0.1:53160)
0927 22:46:20.525415 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.525478 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:20.528484 (+ 3006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:20.529496 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:20.529509 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:20.530358 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:20.530894 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:20.530922 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:20.532570 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:20.532578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:20.534497 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:20.534516 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:20.535497 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:20.535507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:20.535741 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:20.536560 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:20.536596 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:20.539873 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:20.543430 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:20.543440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:20.543451 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:20.543858 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:20.544461 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:20.544465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:20.544467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:20.544638 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:20.545707 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:20.545713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:20.545914 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 22:46:20.546197 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 22:46:20.546744 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":66,"thread_start_us":458,"threads_started":1}
I20260927 22:46:20.547960 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.524368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.1.53.126:46337)
0927 22:46:20.526620 (+ 2252us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:20.526626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:20.526672 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:46:20.528923 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:20.528930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:20.529212 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:20.529396 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:20.531117 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:20.532409 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:20.534665 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:20.535238 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:20.535378 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:20.540153 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:20.540163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:20.540178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:20.540213 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:20.543280 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:20.544052 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:20.544063 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:20.544068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:20.544332 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:20.544763 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:20.544766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:20.544768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:20.545493 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:20.545742 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:20.546355 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:46:20.546938 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":1993,"thread_start_us":1748,"threads_started":1}
I20260927 22:46:20.551084 24127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:46:20.552203 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.551800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45430)
0927 22:46:20.551967 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.552077 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:46:20.552524 24078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
I20260927 22:46:20.554204 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.553805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45440)
0927 22:46:20.553969 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.554039 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:46:20.554411 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39561: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab.
Entry for principal kudu/127.1.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab.
I20260927 22:46:20.580770 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:39561
--webserver_interface=127.1.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab
--principal=kudu/127.1.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46337,127.1.53.125:39561,127.1.53.124:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:46:20.603781 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } attempt: 1
I20260927 22:46:20.605202 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.604584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45448)
0927 22:46:20.604842 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.604985 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:46:20.606429 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.605938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45452)
0927 22:46:20.606128 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.606242 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:46:20.606613 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39561: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
I20260927 22:46:20.674891 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } attempt: 2
I20260927 22:46:20.676281 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.675677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45454)
0927 22:46:20.675957 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.676070 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:46:20.677423 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.676988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45464)
0927 22:46:20.677142 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.677254 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:46:20.677621 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39561: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
I20260927 22:46:20.768932 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } attempt: 3
I20260927 22:46:20.770308 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.769738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45468)
0927 22:46:20.769999 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.770123 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:46:20.771536 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.771105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45484)
0927 22:46:20.771272 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.771362 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 22:46:20.771762 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39561: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
I20260927 22:46:20.915050 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } attempt: 4
I20260927 22:46:20.916419 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.915770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45486)
0927 22:46:20.916107 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.916199 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260927 22:46:20.917718 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:20.917280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45498)
0927 22:46:20.917470 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:20.917589 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 22:46:20.917971 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.125:39561: Network error: Client connection negotiation failed: client connection to 127.1.53.125:39561: connect: Connection refused (error 111)
W20260927 22:46:20.935760 24134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:46:20.936153 24134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:20.936311 24134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:20.936453 24134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:20.943972 24134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:20.944132 24134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:20.944285 24134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:20.944387 24134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:20.944479 24134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:20.954845 24134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46337,127.1.53.125:39561,127.1.53.124:38923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.125.keytab
--principal=kudu/127.1.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.125:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:20.956408 24134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:20.957675 24134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:20.969950 24139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:20.969976 24140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:20.970283 24142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:20.971261 24134 server_base.cc:1034] running on GCE node
I20260927 22:46:20.972013 24134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:20.973577 24134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:20.974825 24134 hybrid_clock.cc:648] HybridClock initialized: now 1790549180974748 us; error 63 us; skew 500 ppm
I20260927 22:46:20.975203 24134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:20 dist-test-slave-pp80 krb5kdc[24047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:20.978405 24134 init.cc:377] Logged in from keytab as kudu/127.1.53.125@KRBTEST.COM (short username kudu)
I20260927 22:46:20.979980 24134 webserver.cc:460] Webserver started at http://127.1.53.125:45343/ using document root <none> and password file <none>
I20260927 22:46:20.980814 24134 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:20.980947 24134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:20.981278 24134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:20.984231 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data/instance:
uuid: "69663ed161cb4dfb90ffbe3f6bea0d74"
format_stamp: "Formatted at 2026-09-27 22:46:20 on dist-test-slave-pp80"
server_key: "ff4e136d1fce5b597442b056d27034df"
server_key_iv: "b90477e4611c1017341a9924b3298960"
server_key_version: "encryptionkey@0"
I20260927 22:46:20.985087 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal/instance:
uuid: "69663ed161cb4dfb90ffbe3f6bea0d74"
format_stamp: "Formatted at 2026-09-27 22:46:20 on dist-test-slave-pp80"
server_key: "ff4e136d1fce5b597442b056d27034df"
server_key_iv: "b90477e4611c1017341a9924b3298960"
server_key_version: "encryptionkey@0"
I20260927 22:46:20.990517 24134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:46:20.994220 24149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:20.995657 24134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:46:20.995863 24134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal
uuid: "69663ed161cb4dfb90ffbe3f6bea0d74"
format_stamp: "Formatted at 2026-09-27 22:46:20 on dist-test-slave-pp80"
server_key: "ff4e136d1fce5b597442b056d27034df"
server_key_iv: "b90477e4611c1017341a9924b3298960"
server_key_version: "encryptionkey@0"
I20260927 22:46:20.996101 24134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:21.021579 24134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:21.025887 24134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:21.026221 24134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:21.045255 24134 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.125:39561
I20260927 22:46:21.045346 24200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.125:39561 every 8 connection(s)
I20260927 22:46:21.046730 24134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/data/info.pb
I20260927 22:46:21.048019 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 24134
I20260927 22:46:21.048280 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-1/wal/instance
I20260927 22:46:21.048833 1237 external_mini_cluster.cc:1428] Setting key d564394735e471735e689a7cf85a1ef5
I20260927 22:46:21.051849 24201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:21.061971 24201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.53.124@KRBTEST.COM; defaulting to no policy
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.53.124@KRBTEST.COM" created.
I20260927 22:46:21.085523 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.064887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.1.53.126:46337)
0927 22:46:21.065410 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.065415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.066594 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 22:46:21.068589 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.068595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.068700 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.068818 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.070454 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.071220 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.072878 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.073302 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.073441 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.077544 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.077550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.077554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.077598 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.081315 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.082490 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.082499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.082505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.082713 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.083268 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.083276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.083282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.083690 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.083856 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.084780 (+ 924us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.085041 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":289,"threads_started":1}
I20260927 22:46:21.086459 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.065598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46337 (local address 127.0.0.1:53162)
0927 22:46:21.066422 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.066462 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.068305 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.069079 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.069088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.069650 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.070237 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.070255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.071398 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.071402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.072723 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.072736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.073539 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.073546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.073800 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.074586 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.074613 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.077242 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.081574 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.081586 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.081609 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.082310 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.082839 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.082846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.082852 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.083129 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.083825 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.083834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.084346 (+ 512us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.084647 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.085186 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
I20260927 22:46:21.088378 24201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:46:21.105952 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.089563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.1.53.125:39561)
0927 22:46:21.090312 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.090323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.090361 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:46:21.090425 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.090437 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.090723 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.090896 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.091593 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.093661 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.094866 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.095428 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.095567 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.099271 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.099284 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.099299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.099333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.102500 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.103485 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.103493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.103495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.103611 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.104113 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.104116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.104118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.104537 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.104729 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.105264 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.105584 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":533,"thread_start_us":380,"threads_started":1}
I20260927 22:46:21.106061 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.089255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45510)
0927 22:46:21.089587 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.089609 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.089727 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.090899 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.090903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.091274 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.091409 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.091417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.093840 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.093844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.094722 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.094733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.095577 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.095583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.095854 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.096519 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.096538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.098974 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.102689 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.102698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.102706 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.103337 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.103760 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.103768 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.103775 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.103984 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.104681 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.104689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.104885 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.105102 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.105459 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":43}
I20260927 22:46:21.108964 24201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:46:21.110308 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.109700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47066)
0927 22:46:21.109940 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.110129 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:46:21.110595 24150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.112186 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.111802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47074)
0927 22:46:21.111972 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.112052 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:46:21.112545 24201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab.
Entry for principal kudu/127.1.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab.
I20260927 22:46:21.115695 1237 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkEfv30/build/asan/bin/kudu
/tmp/dist-test-taskkEfv30/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:38923
--webserver_interface=127.1.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab
--principal=kudu/127.1.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.53.84:43645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.53.126:46337,127.1.53.125:39561,127.1.53.124:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:46:21.161895 24201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 1
I20260927 22:46:21.163095 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.162615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47086)
0927 22:46:21.162821 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.162944 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:46:21.164376 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.163903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47094)
0927 22:46:21.164099 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.164196 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:46:21.164618 24201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.217286 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } attempt: 5
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:46:21.232197 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.218265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.1.53.125:39561)
0927 22:46:21.218479 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.218485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.218508 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:46:21.218594 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.218599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.218693 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.218806 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.219657 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.220791 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.222075 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.222736 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.222865 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.226554 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.226560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.226563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.226596 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.229310 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.230308 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.230313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.230315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.230440 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.230888 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.230892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.230894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.231175 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.231292 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.231799 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.231971 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:46:21.232352 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.218102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45520)
0927 22:46:21.218315 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.218339 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.218467 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.218844 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.218848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.219303 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.219477 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.219486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.220962 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.220966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.221890 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.221904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.222892 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.222896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.223124 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.223838 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.223857 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.226344 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.229512 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.229521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.229528 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.230164 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.230566 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.230574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.230579 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.230782 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.231303 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.231309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.231512 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.231705 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.231989 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:46:21.232852 24201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 2
I20260927 22:46:21.233467 24127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:46:21.233845 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.233403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47108)
0927 22:46:21.233588 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.233691 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:46:21.234589 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.234160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47124)
0927 22:46:21.234338 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.234427 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:46:21.235411 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.235030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47134)
0927 22:46:21.235170 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.235246 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 22:46:21.235611 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.235225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47138)
0927 22:46:21.235387 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.235477 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:46:21.235688 24201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
W20260927 22:46:21.235936 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.287173 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 1
I20260927 22:46:21.288516 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.287968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47144)
0927 22:46:21.288179 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.288344 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:46:21.289620 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.289276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47150)
0927 22:46:21.289439 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.289513 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:46:21.289899 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.326969 24201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 3
I20260927 22:46:21.328171 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.327693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47164)
0927 22:46:21.327901 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.327996 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:46:21.329370 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.328925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47178)
0927 22:46:21.329143 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.329221 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260927 22:46:21.329635 24201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.358189 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 2
I20260927 22:46:21.359398 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.358863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47184)
0927 22:46:21.359100 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.359221 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:46:21.360631 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.360208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47192)
0927 22:46:21.360430 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.360507 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260927 22:46:21.360880 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
W20260927 22:46:21.458024 24209 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:46:21.458474 24209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:46:21.458593 24209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:46:21.458753 24209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:46:21.466450 24209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:46:21.466598 24209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:46:21.466710 24209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:46:21.466797 24209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:46:21.466913 24209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:46:21.467123 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 3
I20260927 22:46:21.468330 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.467771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47204)
0927 22:46:21.467973 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.468075 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:46:21.469394 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.469005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47212)
0927 22:46:21.469228 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.469298 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 22:46:21.469597 24127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.472893 24201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 4
I20260927 22:46:21.473902 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.473461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47216)
0927 22:46:21.473679 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.473756 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:46:21.474866 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.474550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47226)
0927 22:46:21.474692 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.474766 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:46:21.475109 24201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.53.124:38923: Network error: Client connection negotiation failed: client connection to 127.1.53.124:38923: connect: Connection refused (error 111)
I20260927 22:46:21.477855 24209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.53.84:43645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.53.126:46337,127.1.53.125:39561,127.1.53.124:38923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/krb5kdc/kudu_127.1.53.124.keytab
--principal=kudu/127.1.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.53.124:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:46:21.479214 24209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:46:21.480774 24209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:46:21.492635 24215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:21.492669 24214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:46:21.492709 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:46:21.493582 24209 server_base.cc:1034] running on GCE node
I20260927 22:46:21.494400 24209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:46:21.495851 24209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:46:21.497102 24209 hybrid_clock.cc:648] HybridClock initialized: now 1790549181497004 us; error 75 us; skew 500 ppm
I20260927 22:46:21.497464 24209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:46:21.500748 24209 init.cc:377] Logged in from keytab as kudu/127.1.53.124@KRBTEST.COM (short username kudu)
I20260927 22:46:21.502264 24209 webserver.cc:460] Webserver started at http://127.1.53.124:33523/ using document root <none> and password file <none>
I20260927 22:46:21.502933 24209 fs_manager.cc:359] Metadata directory not provided
I20260927 22:46:21.503021 24209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:46:21.503299 24209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:46:21.506508 24209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data/instance:
uuid: "4118ee7e39024227809bcedf23fb9565"
format_stamp: "Formatted at 2026-09-27 22:46:21 on dist-test-slave-pp80"
server_key: "037c425b719d32ceb2675877222a46f0"
server_key_iv: "4e088adf86fef4680925e5254572e14a"
server_key_version: "encryptionkey@0"
I20260927 22:46:21.507287 24209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal/instance:
uuid: "4118ee7e39024227809bcedf23fb9565"
format_stamp: "Formatted at 2026-09-27 22:46:21 on dist-test-slave-pp80"
server_key: "037c425b719d32ceb2675877222a46f0"
server_key_iv: "4e088adf86fef4680925e5254572e14a"
server_key_version: "encryptionkey@0"
I20260927 22:46:21.512909 24209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 22:46:21.516469 24224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:46:21.517853 24209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:46:21.518038 24209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data,/tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal
uuid: "4118ee7e39024227809bcedf23fb9565"
format_stamp: "Formatted at 2026-09-27 22:46:21 on dist-test-slave-pp80"
server_key: "037c425b719d32ceb2675877222a46f0"
server_key_iv: "4e088adf86fef4680925e5254572e14a"
server_key_version: "encryptionkey@0"
I20260927 22:46:21.518273 24209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:46:21.541963 24209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:46:21.547043 24209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:46:21.547353 24209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:46:21.569068 24209 rpc_server.cc:307] RPC server started. Bound to: 127.1.53.124:38923
I20260927 22:46:21.569075 24275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.53.124:38923 every 8 connection(s)
I20260927 22:46:21.570465 24209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/data/info.pb
I20260927 22:46:21.571928 1237 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkEfv30/build/asan/bin/kudu as pid 24209
I20260927 22:46:21.572125 1237 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkEfv30/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790548812741971-1237-0/minicluster-data/master-2/wal/instance
I20260927 22:46:21.572728 1237 external_mini_cluster.cc:1428] Setting key 295668715bb718e4984d725d08006cda
I20260927 22:46:21.575296 24276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:46:21.584335 24276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
I20260927 22:46:21.597245 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.574551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.1.53.124:38923)
0927 22:46:21.575334 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.575368 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.575422 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:46:21.577144 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.577167 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.577556 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.577898 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.579749 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.581300 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.583213 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.583883 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.584010 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.589164 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.589174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.589191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.589234 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.592787 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.593652 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.593661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.593670 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.593889 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.594347 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.594350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.594352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.594870 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.595116 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.595611 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.595992 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":355,"threads_started":1}
I20260927 22:46:21.604272 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.587679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.1.53.126:46337)
0927 22:46:21.588397 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.588403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.588878 (+ 475us) server_negotiation.cc:407] Connection header received
0927 22:46:21.589120 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.589143 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.589226 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.589351 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.590524 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.591653 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.593351 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.593883 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.594015 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.597930 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.597935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.597938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.597973 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.601356 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.602111 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.602116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.602122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.602259 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.602689 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.602692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.602695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.602961 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.603139 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.603597 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.603814 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":280,"threads_started":1}
I20260927 22:46:21.604285 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.587656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.126:46337 (local address 127.0.0.1:53174)
0927 22:46:21.588710 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.588762 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.588965 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.589396 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.589405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.590077 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.590368 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.590377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.591832 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.591837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.593197 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.593213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.594085 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.594091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.594322 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.595026 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.595053 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.597737 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.601534 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.601547 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.601554 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.601978 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.602379 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.602386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.602389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.602524 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.603096 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.603106 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.603305 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.603482 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.603803 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":593,"threads_started":1}
I20260927 22:46:21.605940 24276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.126@KRBTEST.COM
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
I20260927 22:46:21.615792 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.600226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.1.53.126:46337)
0927 22:46:21.600992 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.600996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.601015 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:46:21.601055 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.601058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.601113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.601244 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.602394 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.603213 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.604540 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.605125 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.605266 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.609951 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.609956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.609959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.609990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.612841 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.613582 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.613594 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.613596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.613739 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.614211 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.614216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.614219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.614557 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.614853 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.615060 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.615322 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":408,"threads_started":1}
I20260927 22:46:21.620457 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.606759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.125:39561 (local address 127.0.0.1:45522)
0927 22:46:21.606914 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.606934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.607020 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.607328 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.607331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.607649 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.607778 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.607785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.608929 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.608937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.610196 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.610207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.611147 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.611152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.611274 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.611887 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.611904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.614095 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.617812 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.617819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.617826 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.618301 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.618765 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.618770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.618774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.618926 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.619450 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.619454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.619632 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.619776 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.620063 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:46:21.620558 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.606798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.1.53.125:39561)
0927 22:46:21.607018 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.607024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.607046 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:46:21.607114 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.607119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.607202 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.607320 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.607921 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.608762 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.610335 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.610982 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.611112 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.614316 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.614323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.614328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.614366 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.617635 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.618408 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.618413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.618416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.618597 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.619038 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.619044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.619047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.619327 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.619486 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.619872 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.620112 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:46:21.621521 24276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.53.125@KRBTEST.COM
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549181, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.124@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:46:21.634955 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.617700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.1.53.125:39561)
0927 22:46:21.618261 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.618265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.618285 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:46:21.618323 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.618326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.618413 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.618536 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.619865 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.621036 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.622627 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.623267 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.623398 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.628684 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.628691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.628694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.628728 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.632037 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.632805 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.632811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.632814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.632977 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.633358 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.633364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.633367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.633734 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.633929 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.634314 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.634543 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":261,"threads_started":1}
I20260927 22:46:21.636389 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.622243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47250)
0927 22:46:21.622443 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.622462 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.622566 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.622941 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.622947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.623435 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.623613 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.623622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.624919 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.624922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.625791 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.625802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.626827 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.626833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.627043 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.627659 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.627687 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.630067 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.633557 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.633560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.633563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.634003 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.634428 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.634434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.634437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.634595 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.635222 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.635226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.635357 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.635506 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.635917 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":68}
I20260927 22:46:21.636427 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.622469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.1.53.124:38923)
0927 22:46:21.622642 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.622647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.622672 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:46:21.622724 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.622727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.622778 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.622855 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.623773 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.624788 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.625936 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.626645 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.626756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.630319 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.630326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.630331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.630373 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.633426 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.634123 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.634127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.634129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.634297 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.634720 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.634726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.634732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.634995 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.635180 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.635576 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.636069 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":70}
I20260927 22:46:21.640527 1237 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:46:21.642242 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: Bootstrap starting.
I20260927 22:46:21.642838 24102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c443365fc8bc4966b0507d3151187249"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186
I20260927 22:46:21.645644 24276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:21.646768 24276 log.cc:824] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:21.646821 24127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 4
I20260927 22:46:21.650205 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: No bootstrap required, opened a new log
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.126@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:46:21.659449 24276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.659993 24276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4118ee7e39024227809bcedf23fb9565, State: Initialized, Role: FOLLOWER
I20260927 22:46:21.660617 24276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.661824 24285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.662317 24285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.663208 24276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:21.664695 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.647597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.1.53.124:38923)
0927 22:46:21.647825 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.647833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.647932 (+ 99us) server_negotiation.cc:407] Connection header received
0927 22:46:21.648119 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.648128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.648211 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.648349 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.649288 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.650346 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.652142 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.652962 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.653103 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.657331 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.657338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.657343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.657391 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.661174 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.662158 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.662198 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.662202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.662484 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.662977 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.662983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.662986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.663413 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.663578 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.664049 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.664309 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:46:21.664973 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.647463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47266)
0927 22:46:21.647823 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.647844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.647983 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.648391 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.648396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.648904 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.649109 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.649120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.650479 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.650485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.651979 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.651998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.653117 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.653123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.653335 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.654010 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.654033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.657069 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.661406 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.661412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.661415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.661935 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.662666 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.662672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.662679 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.662858 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.663556 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.663560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.663770 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.663942 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.664315 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260927 22:46:21.671111 24127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249: Bootstrap starting.
I20260927 22:46:21.675457 24127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:21.676823 24127 log.cc:824] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249: Log is configured to *not* fsync() on all Append() calls
W20260927 22:46:21.677115 24296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:46:21.677305 24296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:46:21.680742 24127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249: No bootstrap required, opened a new log
I20260927 22:46:21.674325 24209 master_runner.cc:186] Error getting master registration for 127.1.53.126:46337: OK, instance_id { permanent_uuid: "c443365fc8bc4966b0507d3151187249" instance_seqno: 1790549180502387 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:46:21.685846 24209 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:46:21.687882 24127 raft_consensus.cc:348] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.688301 24127 raft_consensus.cc:729] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c443365fc8bc4966b0507d3151187249, State: Initialized, Role: FOLLOWER
I20260927 22:46:21.688834 24127 consensus_queue.cc:260] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.689836 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.690483 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.691114 24127 sys_catalog.cc:564] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:21.691232 24102 raft_consensus.cc:482] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:46:21.691423 24102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:21.695631 24102 raft_consensus.cc:504] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.699112 24102 leader_election.cc:288] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [CANDIDATE]: Term 1 election: Requested vote from peers 69663ed161cb4dfb90ffbe3f6bea0d74 (127.1.53.125:39561), 4118ee7e39024227809bcedf23fb9565 (127.1.53.124:38923)
I20260927 22:46:21.699239 24251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c443365fc8bc4966b0507d3151187249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4118ee7e39024227809bcedf23fb9565"
I20260927 22:46:21.699889 24251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:21.704515 24251 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c443365fc8bc4966b0507d3151187249 in term 1.
I20260927 22:46:21.705494 24076 leader_election.cc:302] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4118ee7e39024227809bcedf23fb9565, c443365fc8bc4966b0507d3151187249; no voters:
I20260927 22:46:21.706197 24299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:46:21.718868 24299 raft_consensus.cc:686] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [term 1 LEADER]: Becoming Leader. State: Replica: c443365fc8bc4966b0507d3151187249, State: Running, Role: LEADER
I20260927 22:46:21.719756 24299 consensus_queue.cc:237] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.731542 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c443365fc8bc4966b0507d3151187249. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.732111 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:21.738924 24319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 22:46:21.741748 24081 tablet.cc:2369] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:46:21.759075 24319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:46:21.767247 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.729377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.1.53.124:38923)
0927 22:46:21.729614 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.729623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.745047 (+ 15424us) server_negotiation.cc:407] Connection header received
0927 22:46:21.745507 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.745513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.745612 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.745684 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.748116 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.749208 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.751428 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.752132 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.752270 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.756176 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.756185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.756191 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.756264 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.760370 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.762022 (+ 1652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.762035 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.762039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.762434 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.763372 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.763380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.763383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.764117 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.764639 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.765714 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.766362 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:46:21.774307 24201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } attempt: 5
Sep 27 22:46:21 dist-test-slave-pp80 krb5kdc[24047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549180, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.53.125@KRBTEST.COM for kudu/127.1.53.124@KRBTEST.COM
I20260927 22:46:21.785543 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.727354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.1.53.125:39561)
0927 22:46:21.727545 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.727553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.741630 (+ 14077us) server_negotiation.cc:407] Connection header received
0927 22:46:21.741974 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.741980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.742086 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.742167 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.748757 (+ 6590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.749768 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.752758 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.753518 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.753635 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.760414 (+ 6779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.760425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.760432 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.760491 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.764423 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.765332 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.765340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.765350 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.765509 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.771376 (+ 5867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.771382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.771385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.771811 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.772075 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.773470 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.773792 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:46:21.776867 24060 master_runner.cc:186] Error getting master registration for 127.1.53.125:39561: OK, instance_id { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" instance_seqno: 1790549181039816 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:46:21.794243 24060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:46:21.795692 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.774881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.53.124:38923 (local address 127.0.0.1:47292)
0927 22:46:21.775310 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 22:46:21.775335 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:46:21.775450 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:46:21.776468 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:46:21.776477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:46:21.776989 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:46:21.777200 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.777216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.778556 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.778562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.780322 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:46:21.780342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.781771 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.781781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:46:21.782051 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.782850 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:46:21.782879 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:46:21.786732 (+ 3853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:46:21.792086 (+ 5354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.792095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.792101 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.792786 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.793355 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:46:21.793362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:46:21.793368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:46:21.793561 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:46:21.794377 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:46:21.794384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:46:21.794593 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 22:46:21.794765 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:46:21.795162 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
W20260927 22:46:21.800698 24230 tablet.cc:2369] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:46:21.801709 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.775671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.1.53.124:38923)
0927 22:46:21.775887 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.775896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.775913 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:46:21.775977 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.775985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.776087 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.776196 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.777389 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.778380 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.780523 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.781380 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.781525 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.787011 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.787021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.787029 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.787096 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.791788 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.792973 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.792981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.792986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.793194 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.793708 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.793716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.793721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.794154 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.794314 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.796181 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.796567 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:46:21.809494 24201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: Bootstrap starting.
I20260927 22:46:21.815117 24201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: Neither blocks nor log segments found. Creating new log.
I20260927 22:46:21.816892 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:21.726129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.1.53.126:46337)
0927 22:46:21.726408 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:21.726417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:21.727471 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 22:46:21.734494 (+ 7023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:21.734503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:21.734631 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:21.734872 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:21.744988 (+ 10116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.746137 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.751580 (+ 5443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:21.752386 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:21.752518 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:21.766039 (+ 13521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:46:21.766047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:46:21.766052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:46:21.766118 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:46:21.770009 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.797950 (+ 27941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.797961 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.797967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.798296 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:46:21.798953 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:46:21.798960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:46:21.798966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:46:21.799441 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:46:21.799761 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:21.815872 (+ 16111us) server_negotiation.cc:299] Negotiation successful
0927 22:46:21.816311 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:46:21.824488 24201 log.cc:824] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: Log is configured to *not* fsync() on all Append() calls
I20260927 22:46:21.831177 24251 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [term 1 FOLLOWER]: Refusing update from remote peer c443365fc8bc4966b0507d3151187249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:46:21.832430 24309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:46:21.835855 24201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: No bootstrap required, opened a new log
I20260927 22:46:21.850857 24201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.854874 24201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69663ed161cb4dfb90ffbe3f6bea0d74, State: Initialized, Role: FOLLOWER
I20260927 22:46:21.855779 24201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } }
I20260927 22:46:21.859287 24285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c443365fc8bc4966b0507d3151187249. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.859624 24285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.865166 24322 mvcc.cc:205] Tried to move back new op lower bound from 7334089448766799872 to 7334089448332406784. Current Snapshot: MvccSnapshot[applied={T|T < 7334089448766799872}]
I20260927 22:46:21.866050 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.866436 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:21.867524 24326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.869976 24201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:46:21.870082 24176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c443365fc8bc4966b0507d3151187249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74"
I20260927 22:46:21.871698 24175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:46:21.872980 24319 catalog_manager.cc:1347] Generated new cluster ID: e2f72c4db64046eb934e806654e7916a
I20260927 22:46:21.873090 24319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:46:21.875536 24175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [term 1 FOLLOWER]: Refusing update from remote peer c443365fc8bc4966b0507d3151187249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:46:21.876503 24309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:46:21.880489 24176 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c443365fc8bc4966b0507d3151187249 in term 1.
I20260927 22:46:21.882751 24326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.904399 24285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.904722 24285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.909040 24340 mvcc.cc:205] Tried to move back new op lower bound from 7334089448766799872 to 7334089448332406784. Current Snapshot: MvccSnapshot[applied={T|T < 7334089448766799872}]
I20260927 22:46:21.911067 24319 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:46:21.913060 24319 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:46:21.919626 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.920023 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.923454 24326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c443365fc8bc4966b0507d3151187249. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.923894 24326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:46:21.932540 24299 sys_catalog.cc:455] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c443365fc8bc4966b0507d3151187249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c443365fc8bc4966b0507d3151187249" member_type: VOTER last_known_addr { host: "127.1.53.126" port: 46337 } } peers { permanent_uuid: "69663ed161cb4dfb90ffbe3f6bea0d74" member_type: VOTER last_known_addr { host: "127.1.53.125" port: 39561 } } peers { permanent_uuid: "4118ee7e39024227809bcedf23fb9565" member_type: VOTER last_known_addr { host: "127.1.53.124" port: 38923 } } }
I20260927 22:46:21.932883 24299 sys_catalog.cc:458] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249 [sys.catalog]: This master's current role is: LEADER
I20260927 22:46:21.936074 24343 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:46:21.952493 24319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c443365fc8bc4966b0507d3151187249: Generated new TSK 0
I20260927 22:46:21.953755 24319 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:46:22.043815 24282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.031173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.1.53.126:46337)
0927 22:46:22.031552 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.031559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.041035 (+ 9476us) server_negotiation.cc:407] Connection header received
0927 22:46:22.041422 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.041432 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.041553 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.041651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.043387 (+ 1736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53212: BlockingRecv error: recv got EOF from 127.0.0.1:53212 (error 108)
Metrics: {"server-negotiator.queue_time_us":246}
W20260927 22:46:22.044483 24279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.031357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.1.53.124:38923)
0927 22:46:22.031728 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.031734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.042389 (+ 10655us) server_negotiation.cc:407] Connection header received
0927 22:46:22.042872 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.042879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.042987 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.043074 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.044215 (+ 1141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47300: BlockingRecv error: recv got EOF from 127.0.0.1:47300 (error 108)
Metrics: {"server-negotiator.queue_time_us":221}
W20260927 22:46:22.046851 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.031652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.1.53.125:39561)
0927 22:46:22.031885 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.031891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.045078 (+ 13187us) server_negotiation.cc:407] Connection header received
0927 22:46:22.045333 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.045338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.045423 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.045494 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.046603 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45548: BlockingRecv error: recv got EOF from 127.0.0.1:45548 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260927 22:46:22.082583 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.079582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.1.53.125:39561)
0927 22:46:22.079764 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.079771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.080796 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 22:46:22.081054 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.081060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.081209 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.081328 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.082211 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45550: BlockingRecv error: recv got EOF from 127.0.0.1:45550 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 22:46:22.096712 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.094388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.1.53.125:39561)
0927 22:46:22.094585 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.094592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.095160 (+ 568us) server_negotiation.cc:407] Connection header received
0927 22:46:22.095363 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.095368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.095455 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.095563 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.096457 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45552: BlockingRecv error: recv got EOF from 127.0.0.1:45552 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 22:46:22.108450 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.106238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.1.53.125:39561)
0927 22:46:22.106418 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.106425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.106915 (+ 490us) server_negotiation.cc:407] Connection header received
0927 22:46:22.107117 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.107125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.107227 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.107352 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.108169 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45556: BlockingRecv error: recv got EOF from 127.0.0.1:45556 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 22:46:22.120620 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.118277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.1.53.125:39561)
0927 22:46:22.118512 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.118519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.119039 (+ 520us) server_negotiation.cc:407] Connection header received
0927 22:46:22.119262 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.119268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.119363 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.119478 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.120362 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45570: BlockingRecv error: recv got EOF from 127.0.0.1:45570 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260927 22:46:22.134438 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.132139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.1.53.125:39561)
0927 22:46:22.132373 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.132380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.132898 (+ 518us) server_negotiation.cc:407] Connection header received
0927 22:46:22.133103 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.133112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.133231 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.133353 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.134182 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45580: BlockingRecv error: recv got EOF from 127.0.0.1:45580 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 22:46:22.160741 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.158354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.1.53.125:39561)
0927 22:46:22.158613 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.158622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.159172 (+ 550us) server_negotiation.cc:407] Connection header received
0927 22:46:22.159404 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.159410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.159499 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.159613 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.160446 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45584: BlockingRecv error: recv got EOF from 127.0.0.1:45584 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 22:46:22.198686 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.196339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.1.53.125:39561)
0927 22:46:22.196581 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.196590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.197194 (+ 604us) server_negotiation.cc:407] Connection header received
0927 22:46:22.197437 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.197442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.197514 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.197640 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.198440 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45594: BlockingRecv error: recv got EOF from 127.0.0.1:45594 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:46:22.270234 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.267720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.1.53.125:39561)
0927 22:46:22.267964 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.267973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.268622 (+ 649us) server_negotiation.cc:407] Connection header received
0927 22:46:22.268877 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.268883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.268988 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.269108 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.269951 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45596: BlockingRecv error: recv got EOF from 127.0.0.1:45596 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 22:46:22.403973 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.402185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.1.53.125:39561)
0927 22:46:22.402397 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.402401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.402716 (+ 315us) server_negotiation.cc:407] Connection header received
0927 22:46:22.402883 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.402887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.402979 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.403080 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.403802 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36046: BlockingRecv error: recv got EOF from 127.0.0.1:36046 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 22:46:22.666245 24283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:46:22.663989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.1.53.125:39561)
0927 22:46:22.664218 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:22.664226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:22.664594 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:46:22.664836 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:22.664840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:22.664917 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:22.665011 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:46:22.665962 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36062: BlockingRecv error: recv got EOF from 127.0.0.1:36062 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:46:22.679306 24296 catalog_manager.cc:1259] Loaded cluster ID: e2f72c4db64046eb934e806654e7916a
I20260927 22:46:22.679432 24296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: loading cluster ID for follower catalog manager: success
I20260927 22:46:22.681958 24296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: acquiring CA information for follower catalog manager: success
I20260927 22:46:22.683902 24296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4118ee7e39024227809bcedf23fb9565: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:46:22.940519 24343 catalog_manager.cc:1259] Loaded cluster ID: e2f72c4db64046eb934e806654e7916a
I20260927 22:46:22.940647 24343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: loading cluster ID for follower catalog manager: success
I20260927 22:46:22.943315 24343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: acquiring CA information for follower catalog manager: success
I20260927 22:46:22.945334 24343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69663ed161cb4dfb90ffbe3f6bea0d74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:46:23.196313 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:23.186111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.1.53.125:39561)
0927 22:46:23.188865 (+ 2754us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:23.188871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:23.189200 (+ 329us) server_negotiation.cc:407] Connection header received
0927 22:46:23.189426 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:23.189431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:23.189592 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:23.189718 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:23.190295 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.191194 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.193364 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.193859 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.193966 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:23.194669 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:23.195312 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:23.195489 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:23.195750 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:46:23.195832 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2602,"thread_start_us":2378,"threads_started":1}
I20260927 22:46:23.215952 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:23.203527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.1.53.124:38923)
0927 22:46:23.208744 (+ 5217us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:23.208753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:23.208783 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:46:23.208924 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:23.208931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:23.209189 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:23.209444 (+ 255us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:23.210008 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.211142 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.212693 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.213372 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.213542 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:23.214108 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:23.214930 (+ 822us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:23.215147 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:23.215380 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:46:23.215483 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5043,"thread_start_us":4896,"threads_started":1}
I20260927 22:46:23.247932 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:23.226086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.1.53.124:38923)
0927 22:46:23.230536 (+ 4450us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:23.230542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:23.236351 (+ 5809us) server_negotiation.cc:407] Connection header received
0927 22:46:23.236669 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:23.236675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:23.236780 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:23.236853 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:23.237795 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.238904 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.245509 (+ 6605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.246237 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.246350 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:23.247090 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:23.247229 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:23.247289 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:23.247532 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:46:23.247620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4282}
I20260927 22:46:23.248746 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:23.225669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.1.53.126:46337)
0927 22:46:23.235089 (+ 9420us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:23.235097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:23.235609 (+ 512us) server_negotiation.cc:407] Connection header received
0927 22:46:23.235895 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:23.235905 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:23.236213 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:23.237497 (+ 1284us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:23.237724 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.239212 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.245563 (+ 6351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.246324 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.247015 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:23.247218 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:23.247720 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:23.248311 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:23.248366 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 22:46:23.248455 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9247,"thread_start_us":9116,"threads_started":1}
I20260927 22:46:23.251511 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:46:23.225947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.1.53.125:39561)
0927 22:46:23.226375 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 22:46:23.226390 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:46:23.244705 (+ 18315us) server_negotiation.cc:407] Connection header received
0927 22:46:23.245014 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:46:23.245021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:46:23.245120 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:46:23.245248 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:46:23.246001 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.247159 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.249352 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:46:23.250001 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:46:23.250119 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:46:23.250750 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:46:23.250870 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:46:23.250983 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:46:23.251103 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:46:23.251192 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260927 22:46:23.254372 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 24060
I20260927 22:46:23.256151 24122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:23.370048 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 24060
W20260927 22:46:23.391323 24226 connection.cc:535] server connection from 127.0.0.1:47266 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:46:23.391673 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 24134
I20260927 22:46:23.392920 24196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:23.501645 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 24134
I20260927 22:46:23.522506 1237 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskkEfv30/build/asan/bin/kudu pid 24209
I20260927 22:46:23.523744 24271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:46:23.631810 1237 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkEfv30/build/asan/bin/kudu with pid 24209
2026-09-27T22:46:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4992 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9520 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (370852 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST